-- Logs begin at Sat 2021-12-11 20:01:09 UTC, end at Sat 2021-12-11 22:14:52 UTC. -- Dec 11 20:38:51.784201 ncdv-master-1 systemd[1]: Started Devstack devstack@c-sch.service. Dec 11 20:38:55.157160 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:38:55.537848 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Dec 11 20:38:55.541570 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-volume RPC version 3.16 as minimum service version. Dec 11 20:38:55.550849 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:38:55.561747 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Dec 11 20:38:55.565718 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-backup RPC version 2.2 as minimum service version. Dec 11 20:38:55.570311 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Dec 11 20:38:55.576586 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.rpc [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Dec 11 20:38:55.583297 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:38:55.611096 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_concurrency.lockutils [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Acquired lock "singleton_lock" {{(pid=133283) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Dec 11 20:38:55.612086 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_concurrency.lockutils [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Releasing lock "singleton_lock" {{(pid=133283) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Dec 11 20:38:55.613580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ******************************************************************************** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Dec 11 20:38:55.614277 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Configuration options gathered from: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Dec 11 20:38:55.614842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Dec 11 20:38:55.615462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Dec 11 20:38:55.616286 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ================================================================================ {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Dec 11 20:38:55.617060 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allocated_capacity_weight_multiplier = -1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.617639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allow_availability_zone_fallback = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.618163 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allow_compression_on_image_upload = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.618766 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allowed_direct_url_schemes = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.619347 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] api_rate_limit = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.619891 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] auth_strategy = keystone {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.620412 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] az_cache_duration = 3600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.620921 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backdoor_port = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.621423 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backdoor_socket = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.622027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.622601 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_api_class = cinder.backup.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.623131 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.623730 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_name_template = backup-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.624333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_use_same_host = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.624900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_use_temp_snapshot = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.625430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] capacity_weight_multiplier = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.626094 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chap_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.626758 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chap_username = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.627287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.627816 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_internal_tenant_project_id = 0fa26583c3854499bf9a64bce9b104c1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.628368 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_internal_tenant_user_id = 206c81b5fd6548d9894b69b752029642 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.628933 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cloned_volume_same_az = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.629459 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] compression_format = gzip {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.630039 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] compute_api_class = cinder.compute.nova.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.631209 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_dir = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.631789 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.632317 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_source = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.633667 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.634434 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] control_exchange = openstack {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.635007 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] debug = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.635532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.636219 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_group_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.637071 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.637619 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_volume_type = ceph {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.638563 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_client_cert = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.639369 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_client_cert_key = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.640159 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_data_namespace = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.640949 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_ssl_cert_path = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.641749 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_ssl_cert_verify = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.642696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.643434 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_force_upload = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.644017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_new_services = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.644611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_unsupported_driver = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.645176 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enabled_backends = ['ceph'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.645691 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enforce_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.647005 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] filter_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.647527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.648078 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_servers = ['http://10.222.0.18/image'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.648601 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_ssl_compression = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.649108 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_ca_certificates_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.649702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_catalog_info = image:glance:publicURL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.650282 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.650890 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.651398 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.651917 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_num_retries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.652498 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_request_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.653016 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] goodness_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.653525 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] graceful_shutdown_timeout = 5 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.654030 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] group_api_class = cinder.group.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.654724 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] host = ncdv-master-1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.655246 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iet_conf = /etc/iet/ietd.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.655777 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_compress_on_upload = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.656272 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_address_space_limit = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.656954 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_cpu_limit = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.657569 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.658093 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_upload_use_cinder_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.658695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_upload_use_internal_tenant = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.659224 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.659731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_max_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.660249 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_max_size_gb = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.660767 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] instance_format = [instance: %(uuid)s] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.661284 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.661851 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_iotype = fileio {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.662512 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_secondary_ip_addresses = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.663033 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_target_flags = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.663570 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_write_cache = on {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.664099 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.664843 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.665360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.665891 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.667306 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_config_append = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.667840 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.668353 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_dir = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.668884 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.669407 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_options = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.669928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotate_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.670532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotate_interval_type = days {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.671057 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotation_type = none {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.671535 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.671997 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.672615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.673138 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.673595 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.674117 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_age = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.674753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_logfile_count = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.675299 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_logfile_size_mb = 200 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.675826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_over_subscription_ratio = 20.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.676338 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] message_reap_interval = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.676867 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] message_ttl = 2592000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.678389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] monkey_patch = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.678958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] monkey_patch_modules = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.679424 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] my_ip = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.679972 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] no_snapshot_gb_quota = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.680487 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_iser_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.681001 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_shell_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.681534 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_volume_device_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.682047 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nvmet_ns_id = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.682683 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nvmet_port_id = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.683217 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_max_limit = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.683744 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_ext_list = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.684293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.684932 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_listen = 0.0.0.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.685455 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_listen_port = 8776 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.685965 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.686567 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_workers = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.687078 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] per_volume_size_limit = -1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.687754 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] periodic_fuzzy_delay = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.688266 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] periodic_interval = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.688770 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] public_endpoint = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.689282 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] publish_errors = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.689793 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_backup_gigabytes = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.690315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_backups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.690870 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_consistencygroups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.691437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.691946 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_gigabytes = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.692458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_groups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.693590 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_snapshots = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.694259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_volumes = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.694906 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_burst = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.695431 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_except_level = CRITICAL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.695955 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_interval = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.696463 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] replication_device = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.696984 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] report_discard_supported = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.697673 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] report_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.698279 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reservation_clean_interval = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.698788 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reservation_expire = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.699295 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reserved_percentage = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.700020 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.700656 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.701184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rpc_response_timeout = 120 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.701709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] run_external_periodic_tasks = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.702470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_default_filters = ['AvailabilityZoneFilter', 'CapacityFilter', 'CapabilitiesFilter'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.703208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_default_weighers = ['CapacityWeigher'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.703818 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_driver = cinder.scheduler.filter_scheduler.FilterScheduler {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.704340 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_driver_init_wait_time = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.705097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_host_manager = cinder.scheduler.host_manager.HostManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.705698 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_json_config_location = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.706230 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.706790 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_max_attempts = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.707370 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_weight_handler = cinder.scheduler.weights.OrderedHostWeightHandler {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.708056 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scst_target_driver = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.709099 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scst_target_iqn_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.709621 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_down_time = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.710144 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] snapshot_name_template = snapshot-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] snapshot_same_host = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] state_path = /opt/stack/data/cinder {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] storage_availability_zone = nova {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] storage_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] syslog_log_facility = LOG_USER {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.717877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_helper = lioadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.718463 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.718463 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.719742 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.720151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.720535 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] trace_flags = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.720935 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] transfer_api_class = cinder.transfer.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.721333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] transport_url = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.721714 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] until_refresh = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.722097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_chap_auth = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.722521 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_default_quota_class = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.722910 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_eventlog = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.723394 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_journal = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.723779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_json = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.724189 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.724567 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_stderr = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.724944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_syslog = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.725336 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] verify_glance_signatures = enabled {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.725726 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_api_class = cinder.volume.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.726097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_backend_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.726501 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear = zero {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.726885 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear_ionice = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.727260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear_size = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.727650 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.728026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_copy_bps_limit = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.728415 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_dd_blocksize = 1M {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.728811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.729198 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_name_template = volume-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.729581 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_number_multiplier = -1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.729970 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_usage_audit_period = month {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.730436 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.731542 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] watch_log_file = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.731966 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.732360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.732760 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.enforce_new_defaults = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.733236 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.enforce_scope = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.733650 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_default_rule = default {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.734067 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.734467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_file = policy.yaml {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.734883 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.735261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.735639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.736009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.736393 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.736810 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_concurrency.disable_process_locking = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.737191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.737582 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.737981 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.738355 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.738774 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.739156 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.739583 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.connection_string = messaging:// {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.739971 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.740365 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_doc_type = notification {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.740763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_scroll_size = 10000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.741141 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_scroll_time = 2m {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.741520 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.filter_error_trace = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.741909 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.hmac_keys = SECRET_KEY {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.745196 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.sentinel_service_name = mymaster {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.745196 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.socket_timeout = 0.1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.745196 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.trace_sqlalchemy = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.745196 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.745594 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.747702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_token = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.747702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.747702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_admin_prefix = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.747702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.747702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_port = 35357 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.748071 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_protocol = https {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.750660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.750660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_type = password {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.750660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.750660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_version = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.750660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.cache = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.762945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.763398 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.763945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.764375 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.764771 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.765171 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.765558 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.identity_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.765950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.include_service_catalog = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.766343 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.766803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.interface = public {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.774055 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.774505 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.774918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.775321 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.775933 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.776330 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.776763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.777215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.777606 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.778068 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.778506 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.region_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.778957 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.779346 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.779880 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.780309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.780700 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.784231 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.backend = dogpile.cache.null {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.784758 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.backend_argument = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.785195 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.config_prefix = cache.oslo {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.785646 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.debug_cache_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.786045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.786521 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.expiration_time = 600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.788514 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_dead_retry = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.789111 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.789526 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.789962 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_maxsize = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.790359 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.792943 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.793367 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_socket_timeout = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.793813 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.proxies = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.794936 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_allowed_ciphers = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.795335 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_cafile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.795773 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.796213 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.796651 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.797076 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] coordination.backend_url = etcd3+http://10.222.0.18:2379 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.803273 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.backend = sqlalchemy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.803751 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.804316 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_debug = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.804739 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_parameters = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.805180 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_recycle_time = 3600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.805580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_trace = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.806016 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_inc_retry_interval = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.809244 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_max_retries = 20 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.809693 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_max_retry_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.810154 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_retry_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.810603 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_overflow = 50 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.811075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_pool_size = 5 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.811549 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_retries = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.814350 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.mysql_enable_ndb = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.814926 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.815320 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.pool_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.815775 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.retry_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.816167 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.slave_connection = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.816605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.sqlite_synchronous = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.818649 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.use_db_reconnect = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.819084 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.capabilities = [21] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.819480 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.group = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.819865 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.helper_command = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.820260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.820654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.821033 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.821435 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.821810 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.group = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.822187 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.helper_command = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.828051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.828496 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.828899 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.829308 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.829701 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.auth_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.830087 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.cafile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.830486 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.833618 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.collect_timing = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.834028 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.840900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.841366 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.send_service_user_token = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.841779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.842174 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.842888 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.backend_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.843311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chap_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.843794 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chap_username = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.844442 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.845130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_client_cert = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.845636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_client_cert_key = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.846137 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_data_namespace = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.846662 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.847169 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.847753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.848417 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.854898 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.855580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.filter_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.856246 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.goodness_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.856740 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.857156 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.857555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.857947 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.858341 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.858780 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.859192 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_iotype = fileio {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.859593 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.859997 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_target_flags = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.860400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_write_cache = on {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.860803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.861339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.861731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.862129 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.862555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.862948 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.863335 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_shell_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.863720 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.864109 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.864492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.875291 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.replication_device = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.875814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.report_discard_supported = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.876232 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.reserved_percentage = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.876652 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.877045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.877454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.storage_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.877861 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.878401 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.878796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.879201 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.879607 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.879996 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.trace_flags = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.880455 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.unique_fqdn_network = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.881315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.use_chap_auth = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.881703 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.882084 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_backend_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.882758 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear = zero {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.882884 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear_ionice = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.883259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear_size = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.883658 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.884041 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.884438 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.884928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.885319 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.file_event_handler = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.894887 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.895299 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.log_dir = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.899684 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.899934 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.900337 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.900741 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.901136 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.901527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.901921 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.902309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.902722 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.903106 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.903732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.903904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.904305 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.904703 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.905084 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.905490 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.910076 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.910602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.911010 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.911414 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.911856 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.912260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.912672 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.913079 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.913486 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.914125 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_version = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.914292 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.driver = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.914706 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.retry = -1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.915131 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.915517 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.926419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.926661 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.auth_type = password {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.927082 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.928012 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.928414 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.collect_timing = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.928811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.929217 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.interface = public {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.929601 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.930001 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.region_name = RegionOne {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.930394 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.930799 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.931182 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.token_auth_url = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:55.931499 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ******************************************************************************** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Dec 11 20:38:55.931945 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Full set of CONF: {{(pid=133283) _wait_for_exit_or_signal /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:362}} Dec 11 20:38:55.932245 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ******************************************************************************** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Dec 11 20:38:55.932540 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] Configuration options gathered from: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Dec 11 20:38:55.932848 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Dec 11 20:38:55.933156 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Dec 11 20:38:55.946390 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ================================================================================ {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Dec 11 20:38:55.947318 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allocated_capacity_weight_multiplier = -1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.947654 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allow_availability_zone_fallback = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.947968 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allow_compression_on_image_upload = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.948282 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] allowed_direct_url_schemes = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.948741 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] api_rate_limit = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.949047 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] auth_strategy = keystone {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.949536 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] az_cache_duration = 3600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.949988 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backdoor_port = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.950656 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backdoor_socket = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.951109 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.951558 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_api_class = cinder.backup.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.952004 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.952469 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_name_template = backup-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.952913 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_use_same_host = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.953356 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backup_use_temp_snapshot = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.953809 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] capacity_weight_multiplier = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.954262 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chap_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.954750 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chap_username = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.955191 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.955634 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_internal_tenant_project_id = 0fa26583c3854499bf9a64bce9b104c1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.956074 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_internal_tenant_user_id = 206c81b5fd6548d9894b69b752029642 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.956626 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cloned_volume_same_az = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.957085 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] compression_format = gzip {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.957528 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] compute_api_class = cinder.compute.nova.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.957970 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_dir = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.958452 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.958917 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] config_source = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.959358 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.959814 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] control_exchange = openstack {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.961395 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] debug = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.961853 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.962305 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_group_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.966837 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.967348 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] default_volume_type = ceph {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.968140 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_client_cert = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.968599 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_client_cert_key = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.969063 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_data_namespace = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.969516 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_ssl_cert_path = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.969976 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_ssl_cert_verify = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.970430 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] driver_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.970898 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_force_upload = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.971361 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_new_services = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.971816 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enable_unsupported_driver = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.976283 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enabled_backends = ['ceph'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.976835 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] enforce_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.977300 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] filter_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.977764 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.978231 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_servers = ['http://10.222.0.18/image'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.978706 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_api_ssl_compression = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.979170 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_ca_certificates_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.979620 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_catalog_info = image:glance:publicURL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.980067 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.980543 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.981001 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.983921 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_num_retries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.984399 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] glance_request_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.984861 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] goodness_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.985320 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] graceful_shutdown_timeout = 5 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.985774 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] group_api_class = cinder.group.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.986323 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] host = ncdv-master-1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.986806 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iet_conf = /etc/iet/ietd.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.996490 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_compress_on_upload = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.997469 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_address_space_limit = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.997940 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_cpu_limit = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.998402 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.998901 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_upload_use_cinder_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.999351 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_upload_use_internal_tenant = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:55.999811 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.000257 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_max_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.000723 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] image_volume_cache_max_size_gb = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.001177 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] instance_format = [instance: %(uuid)s] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.001621 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.002064 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_iotype = fileio {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.002541 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_secondary_ip_addresses = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.002984 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_target_flags = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.003443 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iscsi_write_cache = on {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.003897 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.004347 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.004794 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.005233 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.006316 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_config_append = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.006802 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.007241 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_dir = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.007681 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.008136 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_options = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.008637 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotate_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.009091 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotate_interval_type = days {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.009527 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] log_rotation_type = none {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.010665 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.013662 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.014765 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.015267 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.015884 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.016553 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_age = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.017940 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_logfile_count = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.019109 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_logfile_size_mb = 200 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.020262 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] max_over_subscription_ratio = 20.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.021854 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] message_reap_interval = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.022859 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] message_ttl = 2592000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.024268 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] monkey_patch = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.024830 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] monkey_patch_modules = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.025446 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] my_ip = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.026271 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] no_snapshot_gb_quota = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.033958 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_iser_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.034441 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_shell_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.034969 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] num_volume_device_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.035444 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nvmet_ns_id = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.035909 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nvmet_port_id = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.036373 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_max_limit = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.036962 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_ext_list = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.037412 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.037861 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_listen = 0.0.0.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.038323 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_listen_port = 8776 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.038848 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.039294 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] osapi_volume_workers = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.039738 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] per_volume_size_limit = -1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.040179 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] periodic_fuzzy_delay = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.040631 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] periodic_interval = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.041087 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] public_endpoint = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.041542 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] publish_errors = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.041984 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_backup_gigabytes = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.042425 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_backups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.042909 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_consistencygroups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.043359 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.043798 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_gigabytes = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.044236 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_groups = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.044673 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_snapshots = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.045109 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] quota_volumes = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.045562 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_burst = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.046000 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_except_level = CRITICAL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.046436 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rate_limit_interval = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.047051 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] replication_device = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.047491 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] report_discard_supported = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.047944 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] report_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.048395 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reservation_clean_interval = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.048832 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reservation_expire = 86400 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.049270 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] reserved_percentage = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.049706 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.050301 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.050793 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] rpc_response_timeout = 120 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.051248 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] run_external_periodic_tasks = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.051692 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_default_filters = ['AvailabilityZoneFilter', 'CapacityFilter', 'CapabilitiesFilter'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.052133 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_default_weighers = ['CapacityWeigher'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.053092 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_driver = cinder.scheduler.filter_scheduler.FilterScheduler {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.053548 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_driver_init_wait_time = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.053996 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_host_manager = cinder.scheduler.host_manager.HostManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.054715 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_json_config_location = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.055168 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.055620 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_max_attempts = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.055925 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scheduler_weight_handler = cinder.scheduler.weights.OrderedHostWeightHandler {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.056227 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scst_target_driver = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.056523 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] scst_target_iqn_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.056910 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_down_time = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.057363 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] snapshot_name_template = snapshot-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.057801 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] snapshot_same_host = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.058259 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.058730 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] state_path = /opt/stack/data/cinder {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.059241 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] storage_availability_zone = nova {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.059677 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] storage_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.060129 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] syslog_log_facility = LOG_USER {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.060577 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_helper = lioadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.061023 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.061467 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.061909 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.062353 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] target_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.062814 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] trace_flags = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.063354 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] transfer_api_class = cinder.transfer.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.063793 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] transport_url = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.064238 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] until_refresh = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.064675 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_chap_auth = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.065107 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_default_quota_class = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.065561 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_eventlog = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.066004 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_journal = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.066447 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_json = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.066926 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.067429 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_stderr = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.068517 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] use_syslog = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.068969 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] verify_glance_signatures = enabled {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.069404 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_api_class = cinder.volume.api.API {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.069842 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_backend_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.070276 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear = zero {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.070736 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear_ionice = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.071190 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_clear_size = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.071628 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.072065 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_copy_bps_limit = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.072497 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_dd_blocksize = 1M {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.072933 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.073382 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_name_template = volume-%s {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.073829 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_number_multiplier = -1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.074264 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volume_usage_audit_period = month {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.074720 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.075151 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] watch_log_file = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 11 20:38:56.075630 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.076089 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.076550 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.enforce_new_defaults = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.077097 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.enforce_scope = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.077615 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_default_rule = default {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.078071 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.078537 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.policy_file = policy.yaml {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.078970 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.079415 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.079853 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.080307 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.080766 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.081215 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_concurrency.disable_process_locking = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.081660 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.082109 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.082579 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.083033 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.083497 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.083945 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.084396 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.connection_string = messaging:// {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.084851 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.085306 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_doc_type = notification {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.085754 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_scroll_size = 10000 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.086191 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.es_scroll_time = 2m {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.086665 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.filter_error_trace = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.087257 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.hmac_keys = SECRET_KEY {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.087781 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.sentinel_service_name = mymaster {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.088250 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.socket_timeout = 0.1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.088700 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] profiler.trace_sqlalchemy = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.089171 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.089615 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.090057 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_token = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.090551 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.admin_user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.090994 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_admin_prefix = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.091442 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.091881 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_port = 35357 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.092319 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_protocol = https {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.093027 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.093478 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_type = password {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.093917 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.094355 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.auth_version = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.094821 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.cache = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.095273 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.095723 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.096162 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.delay_auth_decision = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.096621 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.097059 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.http_connect_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.097515 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.098034 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.identity_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.098515 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.include_service_catalog = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.098959 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.099438 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.interface = public {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.099889 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.100341 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.100781 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.101220 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.101662 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.102105 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.102576 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.103034 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.103474 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.103930 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.104370 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.region_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.104820 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.105273 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_token_roles_required = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.105720 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.service_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.106170 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.token_cache_time = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.106655 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.107114 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.backend = dogpile.cache.null {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.107571 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.backend_argument = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.108137 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.config_prefix = cache.oslo {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.108576 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.debug_cache_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.109015 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.109455 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.expiration_time = 600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.109903 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_dead_retry = 300 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.110571 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.111019 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.111482 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_maxsize = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.111932 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.112375 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.112835 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.memcache_socket_timeout = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.113277 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.proxies = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.113728 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_allowed_ciphers = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.114165 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_cafile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.114648 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.115681 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.116125 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cache.tls_keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.116576 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] coordination.backend_url = etcd3+http://10.222.0.18:2379 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.117032 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.backend = sqlalchemy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.117473 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.118192 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_debug = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.118683 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_parameters = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.119141 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_recycle_time = 3600 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.119581 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.connection_trace = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.120056 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_inc_retry_interval = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.120498 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_max_retries = 20 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.120949 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_max_retry_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.121394 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.db_retry_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.121842 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_overflow = 50 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.122293 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_pool_size = 5 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.122752 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.max_retries = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.123312 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.mysql_enable_ndb = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.124046 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.124784 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.pool_timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.125337 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.retry_interval = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.125804 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.slave_connection = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.126252 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.sqlite_synchronous = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.126747 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] database.use_db_reconnect = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.127215 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.capabilities = [21] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.127669 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.group = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.128406 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.helper_command = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.129197 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.129984 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.131365 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] privsep_osbrick.user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.131878 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.132353 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.group = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.132806 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.helper_command = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.133265 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.133931 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.134390 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] cinder_sys_admin.user = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.134875 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.135370 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.auth_type = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.135858 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.cafile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.136317 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.136825 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.collect_timing = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.137281 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.137778 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.138371 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.send_service_user_token = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.138893 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.139373 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] service_user.timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.139853 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.backend_availability_zone = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.140331 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chap_password = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.140784 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chap_username = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.141238 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.141685 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_client_cert = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.142128 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_client_cert_key = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.142679 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_data_namespace = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.143136 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.143587 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.144041 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.driver_use_ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.144491 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.enable_unsupported_driver = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.144942 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.145587 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.filter_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.149136 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.goodness_function = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.149722 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.150376 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.150876 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.151341 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.151797 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.152320 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.152779 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_iotype = fileio {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.153231 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.153687 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_target_flags = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.154135 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iscsi_write_cache = on {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.154632 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.155098 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.155558 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.156003 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.156456 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.156900 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.157358 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_shell_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.158254 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.158835 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.159295 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.nvmet_port_id = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.159742 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.replication_device = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.160202 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.report_discard_supported = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.160655 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.reserved_percentage = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.161108 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.scst_target_driver = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.162069 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.scst_target_iqn_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.162540 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.storage_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.162982 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_helper = tgtadm {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.163438 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_ip_address = 10.222.0.18 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.163890 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_port = 3260 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.164330 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.164770 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.target_protocol = iscsi {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.165220 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.trace_flags = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.165680 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.unique_fqdn_network = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.166122 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.use_chap_auth = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.166596 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.167038 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_backend_name = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.167479 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear = zero {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.167934 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear_ionice = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.168442 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_clear_size = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.168994 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.169445 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.169884 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.170333 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.170826 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.file_event_handler = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.171267 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.171708 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_reports.log_dir = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.172168 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.172612 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.173071 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.173512 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.173953 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.174391 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.174875 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.175344 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.175788 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.176236 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.176679 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.177116 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.178223 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.178701 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.179237 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.179681 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.180121 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.180560 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.181026 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.181477 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.181935 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.182373 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.182851 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.183307 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.183745 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.184193 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_rabbit.ssl_version = {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.184644 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.driver = [] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.185083 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.retry = -1 {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.185544 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.185985 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] oslo_messaging_notifications.transport_url = **** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.186435 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.auth_section = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.186904 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.auth_type = password {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.187345 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.cafile = /opt/stack/data/ca-bundle.pem {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.187788 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.certfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.188251 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.collect_timing = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.188700 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.insecure = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.189238 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.interface = public {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.189678 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.keyfile = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.190118 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.region_name = RegionOne {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.190609 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.split_loggers = False {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.191061 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.timeout = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.191502 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] nova.token_auth_url = None {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 11 20:38:56.191951 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.service [None req-089d21e5-3b75-4072-a3c4-a34aa2b5c8a4 None None] ******************************************************************************** {{(pid=133283) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Dec 11 20:38:56.192573 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.service [-] Starting cinder-scheduler node (version 19.1.0) Dec 11 20:38:56.201076 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.manager [None req-ad9afdf4-09d5-43b2-a732-721f27c3ca56 None None] Initiating service 1 cleanup Dec 11 20:38:56.206406 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.manager [None req-ad9afdf4-09d5-43b2-a732-721f27c3ca56 None None] Service 1 cleanup completed. Dec 11 20:38:56.206892 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-ad9afdf4-09d5-43b2-a732-721f27c3ca56 None None] Creating RPC server for service cinder-scheduler {{(pid=133283) start /opt/stack/cinder/cinder/service.py:231}} Dec 11 20:38:56.212855 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.service [None req-ad9afdf4-09d5-43b2-a732-721f27c3ca56 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=133283) start /opt/stack/cinder/cinder/service.py:237}} Dec 11 20:38:57.737004 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1b01de52-a124-4919-8734-375ba638f75f None None] Received backup service update from ncdv-master-1: {'backend_state': False, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:38:59.865122 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1a026d42-b469-4818-9150-11bd550ff135 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:38:59.878913 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1a026d42-b469-4818-9150-11bd550ff135 None None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 38, 59, 831720)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:39:03.297267 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.periodic_task [None req-70200d8f-c37f-4f79-8e1c-eb628d67887c None None] Running periodic task SchedulerManager._clean_expired_messages {{(pid=133283) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 11 20:39:03.304434 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.message.api [None req-70200d8f-c37f-4f79-8e1c-eb628d67887c None None] Deleted 0 expired messages. Dec 11 20:39:03.304947 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_service.periodic_task [None req-70200d8f-c37f-4f79-8e1c-eb628d67887c None None] Running periodic task SchedulerManager._clean_expired_reservation {{(pid=133283) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 11 20:40:45.790549 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-62c261e7-e033-4087-9e96-95e469380ff4 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:40:55.103484 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-48fdb05b-f682-4f69-ac07-22caec86715e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:41:45.787471 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-388eb455-f9f8-4b6c-98c1-ab96b3a216f3 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:41:55.109787 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-089090a1-dda2-4fde-af84-28c12b99b849 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:42:45.782445 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ca50e28d-5c39-4c39-9c5c-9e3179c62bed None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:42:55.108090 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-271d310c-f12b-4af5-ad3e-4e877c299a3d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:43:45.787176 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d0a6276a-52f3-437e-b417-1c4dab7222c7 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:43:55.107292 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6be28ccc-b072-4ef4-9cf9-26970b130664 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:44:45.793107 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-157dd465-1381-4549-b982-94346459bb31 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:44:55.105370 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-575b3501-4387-4727-beec-ea995efa99a4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:45:45.799633 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5497ad75-ca6a-4dd8-9910-bcbf1ff64b15 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:45:55.115252 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5707662a-2153-456a-8413-7d1e5b24454e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:46:45.806853 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d302a389-e146-433a-9b52-72c3db60d02d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:46:55.125784 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0e862342-900b-4bd5-83c8-a0c8cfc712e7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:47:45.811384 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-252af994-2fdd-4dc1-9bab-2a0fd4fbfb49 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:47:55.202761 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5fcb2a78-f485-4266-9a06-aca10b706c40 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:48:45.810893 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-be0f0163-2a46-4e93-9e3a-5cfaaea3bddf None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:48:54.767230 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Flow 'volume_create_scheduler' (3f82a4a2-3276-419d-84a8-7a229a36a330) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:48:54.771390 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4c97cae0-ee5a-49b8-8e13-f7f3e22d9aa4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:48:54.774451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4c97cae0-ee5a-49b8-8e13-f7f3e22d9aa4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9f8ce43b-7eb5-406f-8ca7-0888d91599b0),volume_id=9f8ce43b-7eb5-406f-8ca7-0888d91599b0,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:48:54.777940 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6ca59752-4b3a-405a-864a-6a3bb28b6606) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:48:54.787899 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:48:54.816895 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 47, 55, 196464)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:48:54.817856 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:48:54.818444 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:48:54.819059 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9f8ce43b-7eb5-406f-8ca7-0888d91599b0) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:48:54.819705 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:47:55.196464 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:48:54.931050 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:48:54.931521 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:48:54.931926 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:47:55.196464] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:48:54.932621 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Weigher CapacityWeigher returned, weigher value is {max: 398.8, min: 398.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:48:54.932977 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:48:54.933637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:48:54.934027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:48:54.933669 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:48:55.113745 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6ca59752-4b3a-405a-864a-6a3bb28b6606) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:48:55.117827 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-03d13359-a278-4865-92d9-d8802fe0684e tempest-VolumesAdminNegativeTest-1681167821 None] Flow 'volume_create_scheduler' (3f82a4a2-3276-419d-84a8-7a229a36a330) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:48:55.195219 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-90618352-f08b-487d-abea-98cc55160ad5 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:49:09.065616 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Flow 'volume_create_scheduler' (8240ac4a-124a-4f97-bcfc-a95d32357589) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:49:09.068131 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fb8f0442-b700-48ef-9253-eeedff2a7f6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:49:09.082794 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fb8f0442-b700-48ef-9253-eeedff2a7f6b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c0cb7ce4-38ac-4075-b795-d05bb7337ff4),volume_id=c0cb7ce4-38ac-4075-b795-d05bb7337ff4,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:49:09.085241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6852d6b1-8f4a-43f0-ad38-c73f0d970aa9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:49:09.099313 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:49:09.130934 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 48, 55, 186879)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:49:09.131573 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:49:09.131957 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:09.132401 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c0cb7ce4-38ac-4075-b795-d05bb7337ff4) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:49:09.132718 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:48:55.186879 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: from cryptography.utils import int_from_bytes Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: from cryptography.utils import int_from_bytes Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:138: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: last_heartbeat = column_property( Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:144: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: num_hosts = column_property( Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: /opt/stack/cinder/cinder/db/sqlalchemy/models.py:151: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery. Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: num_down_hosts = column_property( Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:49:09.136912 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:49:09.138714 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:49:09.141239 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 398.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:49:09.143232 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:49:09.145298 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:49:09.145298 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:49:09.148263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:09.148263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:09.148263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:48:55.186879] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:49:09.148263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Weigher CapacityWeigher returned, weigher value is {max: 398.8, min: 398.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:49:09.148263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:49:09.149156 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:49:09.150277 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:09.148146 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:49:09.212246 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6852d6b1-8f4a-43f0-ad38-c73f0d970aa9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:49:09.226884 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-baec5fb2-7037-4fbe-adc3-7f06019ea0bb tempest-VolumesAdminNegativeTest-1681167821 None] Flow 'volume_create_scheduler' (8240ac4a-124a-4f97-bcfc-a95d32357589) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:49:11.373501 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-453e2acc-d1b8-4873-b5f1-6fb2228c8693 tempest-VolumesAdminNegativeTest-1681167821 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:49:12.739678 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dc2f86de-2b9d-4555-8c02-d89906991d80 tempest-VolumesAdminNegativeTest-1681167821 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:49:37.153979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Flow 'volume_create_scheduler' (554fbc80-9cfa-4e20-99f2-10176b6f80dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:49:37.160436 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cca15259-c72c-4e20-a044-c1e63840846d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:49:37.167149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cca15259-c72c-4e20-a044-c1e63840846d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(02cbc06b-9ea4-4656-a459-2ec9f65983a3),volume_id=02cbc06b-9ea4-4656-a459-2ec9f65983a3,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:49:37.172231 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a1353eaa-b3f8-4c4f-9bb9-cb851fe101f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:49:37.186971 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:49:37.226392 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 49, 12, 731230)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:49:37.227091 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:49:37.227495 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:37.227831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (02cbc06b-9ea4-4656-a459-2ec9f65983a3) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:49:37.228231 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:12.731230 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:49:37.229338 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:49:37.230938 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 398.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:49:37.232218 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:49:37.233921 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:49:37.235345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:37.235345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:49:37.235345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:12.731230] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:49:37.235345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Weigher CapacityWeigher returned, weigher value is {max: 398.8, min: 398.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:49:37.235345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:49:37.235659 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:49:37.235659 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:37.231757 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:49:37.283251 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a1353eaa-b3f8-4c4f-9bb9-cb851fe101f8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:49:37.285429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-68b770e5-e348-4a6d-99d9-f57be5cb8a32 tempest-AttachSCSIVolumeTestJSON-819908471 None] Flow 'volume_create_scheduler' (554fbc80-9cfa-4e20-99f2-10176b6f80dc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:49:45.813685 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf8eb777-a031-4ebb-a439-e0d7330476fc None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:49:55.196797 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-651bac19-6410-40b5-86c5-79a0bf866b89 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:49:58.304881 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c9121cb8-f351-4310-8525-423b7cde0489 tempest-AttachSCSIVolumeTestJSON-819908471 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:50:01.461763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Flow 'volume_create_scheduler' (96acc04b-20c0-4ef7-8716-5c791142ab15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:50:01.466901 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7474d9ec-19ec-447c-a3ac-056f1c887ffb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:50:01.470433 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7474d9ec-19ec-447c-a3ac-056f1c887ffb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0dd079d3-9d32-4641-aba0-15729e22fd77),volume_id=0dd079d3-9d32-4641-aba0-15729e22fd77,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:50:01.481077 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bcb93560-75f0-4ab6-9f5a-3d9b8c44c8e8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:50:01.488983 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:50:01.504432 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 0, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 49, 58, 300413)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:50:01.505127 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:50:01.505650 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:50:01.506172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0dd079d3-9d32-4641-aba0-15729e22fd77) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:50:01.507298 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:58.300413 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:50:01.508559 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 398.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:50:01.510558 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:50:01.512621 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:50:01.514353 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:50:01.516063 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:50:01.516063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:50:01.516063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:50:01.516063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 0, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 0,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:49:58.300413] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:50:01.516063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Weigher CapacityWeigher returned, weigher value is {max: 398.8, min: 398.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:50:01.516389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:50:01.516389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:50:01.516389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:50:01.511025 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:50:01.583188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bcb93560-75f0-4ab6-9f5a-3d9b8c44c8e8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:50:01.591723 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-70990fe6-18ff-477b-8253-e54810a107a4 tempest-LiveMigrationTest-62741823 None] Flow 'volume_create_scheduler' (96acc04b-20c0-4ef7-8716-5c791142ab15) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:50:45.814504 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c6331db2-a3f9-4c8f-b70d-8c1e1fe7b2ca None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:50:55.205920 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b69b658c-4ba0-47db-bc4a-28ddc9d8ee57 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:51:05.071017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Flow 'volume_create_scheduler' (5a13da1e-84d2-44d2-8280-6e1f0ae60b8c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:51:05.077351 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4c394e00-3049-49c3-95a1-dbd36aeba4b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:51:05.082210 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4c394e00-3049-49c3-95a1-dbd36aeba4b3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(dc518867-b86b-43da-a7ed-f4805e120bdf),volume_id=dc518867-b86b-43da-a7ed-f4805e120bdf,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:51:05.085849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (010917d4-38d0-40e7-8b2e-97bb0db1abd3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:51:05.102662 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:51:05.126022 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.94, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 50, 55, 201355)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:51:05.127784 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:51:05.128360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:51:05.128917 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (dc518867-b86b-43da-a7ed-f4805e120bdf) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:51:05.129620 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:50:55.201355 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:51:05.131231 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:51:05.131788 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:51:05.132340 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.94, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:50:55.201355] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:51:05.132937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Weigher CapacityWeigher returned, weigher value is {max: 397.8, min: 397.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:51:05.133453 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:51:05.134302 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:51:05.134873 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.94, total_capacity_gb: 19.94,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:51:05.134505 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:51:05.174003 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (010917d4-38d0-40e7-8b2e-97bb0db1abd3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:51:05.176679 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ca5d290e-fa73-47f8-9c65-223503cb8e79 tempest-LiveMigrationTest-62741823 None] Flow 'volume_create_scheduler' (5a13da1e-84d2-44d2-8280-6e1f0ae60b8c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:51:45.815386 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0b87c63e-55b7-4271-adf6-7d65efc5158d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:51:55.229676 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d2c0856b-7413-42f6-b6c2-7253913543d3 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.5, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 3, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:52:29.620026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-ef9fd12e-71fb-4232-a8aa-2441446def33 req-b592ce28-936b-422c-b294-5032079fcca1 tempest-LiveMigrationTest-62741823 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.5, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 2, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:52:30.906811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e4bf8f32-8e8b-4d9d-8b89-b750cd2a2b99 tempest-LiveMigrationTest-62741823 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:52:45.814107 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-da73b4a6-1351-4ac7-8f5d-f2b52cb58fa0 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:52:55.236170 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6157fc26-5520-40f2-8334-9e99c269f705 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:53:45.818810 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d9f0b83e-2d24-4c2e-b548-805dc3a3ee1e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:53:55.242150 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6c5fc119-1aad-4b97-aa1b-4164f7f220cf None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:54:45.818556 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bacc4083-fb29-439c-b0e8-e7a3c8ced90a None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:54:55.205774 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3b969d98-460f-45c7-8a7b-bef599459138 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:55:00.499460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (dd083b90-d2c6-49b8-93ba-54ddb3baf434) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:00.503378 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (92e71e78-89d3-4bc4-b712-e7189112e90b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:00.512799 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (92e71e78-89d3-4bc4-b712-e7189112e90b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a2771b04-1309-441b-b6e9-2c6b44964ee1),volume_id=a2771b04-1309-441b-b6e9-2c6b44964ee1,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:00.522145 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e7f26ae1-193f-4c3a-b1a1-41641c241959) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:00.535554 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:55:00.584191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 54, 55, 201098)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:55:00.584191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:55:00.584191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:00.584191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a2771b04-1309-441b-b6e9-2c6b44964ee1) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:55:00.585839 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:54:55.201098 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 398.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:55:00.586069 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:55:00.589626 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:55:00.595079 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 398.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:55:00.599082 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 1, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 1,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:54:55.201098] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:55:00.605882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Weigher CapacityWeigher returned, weigher value is {max: 397.8, min: 397.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:55:00.607772 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:55:00.607772 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:55:00.607772 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:00.595808 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:55:00.692475 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e7f26ae1-193f-4c3a-b1a1-41641c241959) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:00.710183 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c486ace6-f75c-468c-b78a-1c99ef587fec tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (dd083b90-d2c6-49b8-93ba-54ddb3baf434) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:03.311842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (fa851e83-87d6-403c-a238-286d8be6468a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:03.322186 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (643394cb-189c-4616-97f0-73bfee99edd6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:03.331114 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (643394cb-189c-4616-97f0-73bfee99edd6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c1dfe182-bb86-46fb-a3e2-803cc6c4178f),volume_id=c1dfe182-bb86-46fb-a3e2-803cc6c4178f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:03.333443 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2c417e3d-b1dd-49b3-96cb-744698b6dfd0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:03.359056 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:55:03.420873 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 54, 55, 201098)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:55:03.422807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:55:03.423182 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:03.423547 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c1dfe182-bb86-46fb-a3e2-803cc6c4178f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:55:03.424046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:00.595808 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:55:03.427706 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:55:03.429922 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 389.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:03.431775 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:55:03.434758 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:55:03.436824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:03.436824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:03.436824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 2, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 2,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:00.595808] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:55:03.436824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Weigher CapacityWeigher returned, weigher value is {max: 396.8, min: 396.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:55:03.436824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:55:03.437339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:55:03.437995 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:03.431836 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:55:03.481595 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2c417e3d-b1dd-49b3-96cb-744698b6dfd0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:03.484042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dc9eb358-3643-4fb0-83ff-db6ce3e94a08 tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (fa851e83-87d6-403c-a238-286d8be6468a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:05.021928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (3c6a604e-fba0-4330-b2d0-66f5f22102e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:05.024825 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (468200f1-7d07-4152-8cd8-b202673d6102) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:05.027908 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (468200f1-7d07-4152-8cd8-b202673d6102) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(063ad756-f8f1-422c-8ef8-c139a6abdaf5),volume_id=063ad756-f8f1-422c-8ef8-c139a6abdaf5,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:05.051771 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5b3102de-9ae5-431a-ab46-0ec00adba4cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:55:05.060686 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:55:05.101840 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 1, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 54, 55, 201098)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:55:05.102176 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:55:05.102565 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:05.102922 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (063ad756-f8f1-422c-8ef8-c139a6abdaf5) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:55:05.103513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:03.431836 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:55:05.104741 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 369.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:05.106750 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:55:05.108996 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:55:05.114273 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:55:05.121148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:05.121148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:55:05.121148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 3, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 3,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:03.431836] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:55:05.121148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Weigher CapacityWeigher returned, weigher value is {max: 395.8, min: 395.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:55:05.121148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:55:05.122149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:55:05.122149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.49, total_capacity_gb: 19.94,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:05.110942 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:55:05.187603 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5b3102de-9ae5-431a-ab46-0ec00adba4cd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:55:05.194915 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1de59102-e6c7-4b17-a887-4ff1a98245fc tempest-TaggedBootDevicesTest-262552579 None] Flow 'volume_create_scheduler' (3c6a604e-fba0-4330-b2d0-66f5f22102e1) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:55:45.824792 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f0454b2e-3067-4638-85d0-c0467c8b8540 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:55:55.241944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9b07ef39-efb4-4dfd-8c71-11ba0dd2e76c None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:56:00.369116 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Flow 'volume_create_scheduler' (e166d1cd-815b-4988-b5d1-83f3e5bc7886) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:00.380723 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (68a3189a-fae1-46d0-8686-d8e3464cf5c7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:00.389873 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (68a3189a-fae1-46d0-8686-d8e3464cf5c7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e515830a-7134-4859-a11a-6f2e811d1996),volume_id=e515830a-7134-4859-a11a-6f2e811d1996,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:00.393458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3385e998-56c5-47f2-b5a3-75b0eb6668af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:00.408839 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:00.442940 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 55, 55, 236887)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:00.443456 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:00.448470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:00.449118 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e515830a-7134-4859-a11a-6f2e811d1996) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:00.449980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:55.236887 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:00.456151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:00.456560 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:00.456948 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:55:55.236887] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:00.458615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Weigher CapacityWeigher returned, weigher value is {max: 394.8, min: 394.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:00.458975 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:00.462795 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:00.463261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:00.462826 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:00.526747 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3385e998-56c5-47f2-b5a3-75b0eb6668af) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:00.528725 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fbd7997b-143c-417f-b4b5-393e87cf263e tempest-LiveAutoBlockMigrationV225Test-197978676 None] Flow 'volume_create_scheduler' (e166d1cd-815b-4988-b5d1-83f3e5bc7886) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:25.325576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (a157976f-d658-4c51-b318-6950ccc56482) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:25.337690 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b001654f-b824-4d7f-8c15-543811776ce0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:25.343750 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b001654f-b824-4d7f-8c15-543811776ce0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f8896d9b-d453-458b-9137-4b5e37666eb4),volume_id=f8896d9b-d453-458b-9137-4b5e37666eb4,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:25.357601 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc4a6318-4c1e-4016-ab6f-9a507086a29d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:25.380332 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:25.397594 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 55, 55, 236887)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:25.398510 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:25.399059 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:25.399600 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f8896d9b-d453-458b-9137-4b5e37666eb4) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:25.400205 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:00.462826 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 349.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:25.405990 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:56:25.408785 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:56:25.411143 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 389.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:25.413213 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:25.419849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:25.419849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:25.419849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:00.462826] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:25.419849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Weigher CapacityWeigher returned, weigher value is {max: 393.8, min: 393.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:25.419849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:25.420341 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:25.420341 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:25.418057 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:25.459693 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fc4a6318-4c1e-4016-ab6f-9a507086a29d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:25.468801 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0c55ccbe-4ef6-4c87-b14f-18a25ffe6c1b tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (a157976f-d658-4c51-b318-6950ccc56482) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:33.137689 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (975efffd-3f05-4c4a-95b3-72aec41e5279) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:33.152414 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b8ecea7-e9c1-4518-81b5-82015ee65d67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:33.155305 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b8ecea7-e9c1-4518-81b5-82015ee65d67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bacff558-89b8-4eb4-a4cc-bdf5b3ea8f07),volume_id=bacff558-89b8-4eb4-a4cc-bdf5b3ea8f07,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:33.157396 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ea37a522-1101-4512-87c3-7af67fad5d4d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:33.170306 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:33.208533 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 55, 55, 236887)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:33.208925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:33.209941 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:33.210314 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (bacff558-89b8-4eb4-a4cc-bdf5b3ea8f07) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:33.210741 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:25.418057 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:56:33.211801 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:56:33.214184 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 369.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:33.216367 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:56:33.219276 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:56:33.223323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:33.223323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:33.223323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:25.418057] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:33.223323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Weigher CapacityWeigher returned, weigher value is {max: 392.8, min: 392.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:33.223323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:33.223882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:33.223882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.49, total_capacity_gb: 19.94,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:33.214160 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:33.250162 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ea37a522-1101-4512-87c3-7af67fad5d4d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:33.254416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-df9e2b88-58b2-4199-b4a7-96efb66a480e tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (975efffd-3f05-4c4a-95b3-72aec41e5279) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:35.684838 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (45b9ba71-056e-4094-8396-d229ded183fd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:35.687709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (01fbb032-394e-4683-8d8e-90264ee1df3b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:35.690195 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (01fbb032-394e-4683-8d8e-90264ee1df3b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1cc24081-5bd3-4779-a26a-13ece3c04557),volume_id=1cc24081-5bd3-4779-a26a-13ece3c04557,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:35.693664 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b01459b1-84d8-4dd2-b1c3-00486d85cdf8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:35.702469 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:35.733835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 55, 55, 236887)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:35.734243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:35.734646 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:35.735012 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1cc24081-5bd3-4779-a26a-13ece3c04557) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:35.735466 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.49, total_capacity_gb: 19.94,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:33.214160 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:56:35.739941 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 349.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:35.742591 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:56:35.745239 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:56:35.747290 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:56:35.751125 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:35.751625 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:35.752293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.49, total_capacity_gb: 19.94,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:33.214160] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:35.752813 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Weigher CapacityWeigher returned, weigher value is {max: 391.8, min: 391.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:35.753431 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:35.753989 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:35.754752 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.489999999999998, total_capacity_gb: 19.94,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:35.754011 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:35.791241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b01459b1-84d8-4dd2-b1c3-00486d85cdf8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:35.794086 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-98367332-38de-447b-8896-04f9bb7fd5de tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (45b9ba71-056e-4094-8396-d229ded183fd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:36.980380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (f1b2a04c-15e0-4fbe-946d-99671912a4c8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:36.982853 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20e0c508-9b58-4e54-912a-0c2c5b8fd60e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:36.985178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20e0c508-9b58-4e54-912a-0c2c5b8fd60e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2598d679-ec5a-4b7f-a18e-b0e2e1dd486c),volume_id=2598d679-ec5a-4b7f-a18e-b0e2e1dd486c,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:36.987150 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d6758063-bf8d-4085-a38e-bb83bdf5546f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:36.994881 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:37.008814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 55, 55, 236887)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:37.009240 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:37.009610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:37.010031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2598d679-ec5a-4b7f-a18e-b0e2e1dd486c) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:37.010356 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.489999999999998, total_capacity_gb: 19.94,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:35.754011 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:37.011768 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:37.012154 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:37.012531 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.489999999999998, total_capacity_gb: 19.94,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:35.754011] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:37.012942 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Weigher CapacityWeigher returned, weigher value is {max: 390.8, min: 390.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:37.013288 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:37.013753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:37.014130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.489999999999998, total_capacity_gb: 19.94,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:37.013783 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:37.041450 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d6758063-bf8d-4085-a38e-bb83bdf5546f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:37.043529 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1d3f6fe6-0a57-4479-8877-b355f94c0b8d tempest-TaggedBootDevicesTest_v242-10492700 None] Flow 'volume_create_scheduler' (f1b2a04c-15e0-4fbe-946d-99671912a4c8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:45.827837 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7d0fe6eb-95f3-4626-817a-c3a2dd54fd1e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:56:55.224042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ff9fafca-3daa-4ffb-9626-dfe5973dc0be None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:56:55.578079 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Flow 'volume_create_scheduler' (fe624137-0b8b-4172-8e86-46c6b5f2e32f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:56:55.591049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (436bed65-584a-405e-842a-dd5c862a1d55) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:55.593475 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (436bed65-584a-405e-842a-dd5c862a1d55) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(be24a1b4-89d0-466f-a8ab-5fe44740018f),volume_id=be24a1b4-89d0-466f-a8ab-5fe44740018f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:55.596060 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (35de824b-a05c-4d05-aa30-a27671f9f296) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:56:55.607516 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:56:55.631516 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 56, 55, 219890)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:56:55.632237 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:56:55.632831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:55.633365 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (be24a1b4-89d0-466f-a8ab-5fe44740018f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:56:55.633969 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:55.219890 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 329.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:55.635234 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:56:55.637924 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:56:55.641180 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 309.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:56:55.643489 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:56:55.650735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:55.650735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:56:55.650735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.49, total_capacity_gb: 19.94,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:55.219890] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:56:55.650735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Weigher CapacityWeigher returned, weigher value is {max: 389.8, min: 389.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:56:55.651721 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:56:55.651721 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:56:55.652009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:55.650175 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:56:55.729248 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (35de824b-a05c-4d05-aa30-a27671f9f296) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:56:55.731762 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3a522c46-b19d-480f-9228-6d10c7d299e4 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Flow 'volume_create_scheduler' (fe624137-0b8b-4172-8e86-46c6b5f2e32f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:57:15.169015 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Flow 'volume_create_scheduler' (4b71f1fa-994f-4e2a-ae1e-956a3e26a11a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:57:15.174381 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (11e69070-765e-4ced-a243-2fdfd8c41580) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:57:15.186453 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (11e69070-765e-4ced-a243-2fdfd8c41580) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(137a90da-6642-41d0-9f99-631c12ed1b96),volume_id=137a90da-6642-41d0-9f99-631c12ed1b96,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:57:15.192307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2a4add7f-5563-4f2d-a8fa-0944335b3689) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:57:15.206021 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:57:15.241766 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 56, 55, 219890)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:57:15.242071 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:57:15.242451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (137a90da-6642-41d0-9f99-631c12ed1b96) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:55.650175 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:57:15.250227 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:57:15.256508 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 389.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:57:15.269662 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:57:15.280731 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:15.287143 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:15.290372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.49, total_capacity_gb: 19.94,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:56:55.650175] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:57:15.290372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Weigher CapacityWeigher returned, weigher value is {max: 388.8, min: 388.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:57:15.290372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:57:15.290372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:57:15.291410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:57:15.248130 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:57:15.299013 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2a4add7f-5563-4f2d-a8fa-0944335b3689) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:57:15.302106 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a9f56a68-f2cc-4e3f-8720-5b0accae192a tempest-ServerRescueNegativeTestJSON-997522025 None] Flow 'volume_create_scheduler' (4b71f1fa-994f-4e2a-ae1e-956a3e26a11a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:57:42.959778 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Flow 'volume_create_scheduler' (7678eeac-ba15-46f4-860a-8d5a9092e187) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:57:42.963548 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a99c972a-b700-45b4-ad20-924fcba0ebb7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:57:42.966175 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a99c972a-b700-45b4-ad20-924fcba0ebb7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(885cbf69-4141-4d54-b3d4-193932c5960f),volume_id=885cbf69-4141-4d54-b3d4-193932c5960f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:57:42.968312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (553c9471-1204-44f1-88f2-c4a36eb567c3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 20:57:42.977227 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 20:57:42.991291 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.94, 'free_capacity_gb': 19.49, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 56, 55, 219890)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 20:57:42.991689 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 20:57:42.992163 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:42.992707 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (885cbf69-4141-4d54-b3d4-193932c5960f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 20:57:42.993324 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:57:15.248130 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:57:42.994407 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 369.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:57:42.995765 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 20:57:42.997229 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 20:57:42.998459 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 20:57:43.000006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:43.000006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 20:57:43.000006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.49, total_capacity_gb: 19.94,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:57:15.248130] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 20:57:43.000006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Weigher CapacityWeigher returned, weigher value is {max: 387.8, min: 387.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 20:57:43.000006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 20:57:43.000697 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 20:57:43.000697 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.49, total_capacity_gb: 19.94,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:57:42.996850 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 20:57:43.031692 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (553c9471-1204-44f1-88f2-c4a36eb567c3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 20:57:43.035773 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9ebbb76e-ff8b-4b29-b10a-b7c6831cbe8d tempest-ServerRescueNegativeTestJSON-997522025 None] Flow 'volume_create_scheduler' (7678eeac-ba15-46f4-860a-8d5a9092e187) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 20:57:44.717837 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-b7924522-8375-4715-8740-f0c9be97e4da req-789263ff-87d9-4025-85d7-1e53a27580e1 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.44, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:57:45.419921 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-579d6526-7609-40df-86da-20d0ff712803 tempest-LiveAutoBlockMigrationV225Test-197978676 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.93, 'free_capacity_gb': 19.44, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:57:45.828660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ae94703e-f6df-4a11-9914-7f3326c8bf7f None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:57:55.121558 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-56f1f6ed-4aef-42d9-bbe5-6880ca6b6a42 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.92, 'free_capacity_gb': 19.42, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:58:30.539075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1a7772a8-b42d-4538-af24-111835e7ef26 tempest-ServerRescueNegativeTestJSON-997522025 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.88, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:58:31.656958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0b47b06c-dbd7-4935-9237-caf42507e04a tempest-ServerRescueNegativeTestJSON-997522025 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.88, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:58:45.833275 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6573f334-5d9f-47f1-818c-98ac569ba7ff None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:58:55.223547 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5c261429-ba8f-4e84-aa68-aa5e9f325ede None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.87, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 20:59:45.838837 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6c0fd76c-6d82-43e3-8459-7ef1071ead02 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 20:59:55.215329 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-11d9156e-9f12-4961-b5e3-46dc00f08403 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.86, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:00:45.837645 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6c716174-3487-4c9e-8b4c-8503a09b6e4d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:00:51.602130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (e64301d3-0a0f-4498-ba3c-ac4341bd8564) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:00:51.608700 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ff363244-c5c5-4283-9ee5-f58c023a11ca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:00:51.612839 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ff363244-c5c5-4283-9ee5-f58c023a11ca) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4b0f1f89-d730-4940-bf8e-78b633d9c4f1),volume_id=4b0f1f89-d730-4940-bf8e-78b633d9c4f1,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:00:51.621821 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (431a69b6-0dbf-46c1-89fb-e1845b2b6d68) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:00:51.645373 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:00:51.671595 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.86, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 20, 59, 55, 209809)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:00:51.672417 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:00:51.673060 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:51.675024 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4b0f1f89-d730-4940-bf8e-78b633d9c4f1) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:00:51.675651 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.86,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:59:55.209809 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:00:51.685566 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:51.689938 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:51.690360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.86,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 20:59:55.209809] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:00:51.690824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Weigher CapacityWeigher returned, weigher value is {max: 389.2, min: 389.2} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:00:51.691203 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:00:51.691661 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:00:51.692046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.86,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:51.691695 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:00:51.747064 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (431a69b6-0dbf-46c1-89fb-e1845b2b6d68) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:00:51.754506 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1f9a4522-0d23-460f-bba5-419bc0fdc58a tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (e64301d3-0a0f-4498-ba3c-ac4341bd8564) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:00:55.236888 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-093945b2-c217-45f5-8305-92ea886c1916 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:00:57.429732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (180dcccd-82ee-42cf-b737-80c55e1c354f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:00:57.433212 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7dbd2bba-605f-4176-b9fd-104a634027d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:00:57.435690 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7dbd2bba-605f-4176-b9fd-104a634027d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(6d218a8d-14e8-4548-bd7c-d1dc9448b64d),volume_id=6d218a8d-14e8-4548-bd7c-d1dc9448b64d,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:00:57.437620 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7a2ba1e5-e1d5-4e84-a31d-b5f7cb74806e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:00:57.445861 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:00:57.464781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 0, 55, 229691)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:00:57.465221 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:00:57.466632 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:57.467114 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (6d218a8d-14e8-4548-bd7c-d1dc9448b64d) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:00:57.467527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:55.229691 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 349.79999999999995, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:00:57.468815 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:00:57.471766 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:00:57.473206 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:00:57.475931 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:55.229691] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:00:57.481379 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:00:57.491032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:00:57.491032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:00:57.491032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:57.472169 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:00:57.516013 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7a2ba1e5-e1d5-4e84-a31d-b5f7cb74806e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:00:57.519135 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b32c914b-a360-4f31-9210-dbaf075a8ef8 tempest-AttachVolumeTestJSON-496084914 None] Flow 'volume_create_scheduler' (180dcccd-82ee-42cf-b737-80c55e1c354f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:12.639892 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 0, 55, 229691)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:12.640524 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:12.640674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.641043 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:12.641429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.644339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.645046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:57.472169] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:01:12.646349 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:12.648123 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:12.669383 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 0, 55, 229691)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:12.669860 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:12.670244 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.670656 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:12.670991 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.672751 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:12.674247 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:57.472169] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:01:12.674854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:12.675912 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:12.677137 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d6eccbf5-da5b-44a8-8b81-662e87d91faa tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:01:14.002405 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 0, 55, 229691)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:14.005556 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:14.006702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.007469 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:14.008017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.008628 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.009243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:57.472169] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:01:14.009924 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:14.010418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:14.019377 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 0, 55, 229691)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:14.020006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:14.020584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.021031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:14.021598 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.022149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:14.022759 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:00:57.472169] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:01:14.023400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:14.023935 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:14.024362 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ba3552ce-e779-48bb-8bda-6281eba9c3ea tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:01:14.331312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-27571857-c731-4762-aef1-fd3d136c1e40 tempest-AttachVolumeTestJSON-496084914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:15.598349 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-67053e10-5f96-4cbe-a91c-c28a447cd7d0 tempest-AttachVolumeTestJSON-496084914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:15.697310 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (166e0e14-efe7-4e34-9d0d-56586f432bee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:15.705252 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b6cd5242-f729-4eaf-9a47-4fd6b398ce6b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:15.713459 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b6cd5242-f729-4eaf-9a47-4fd6b398ce6b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=0466b59d-16f3-4595-a02f-2847d0e46780,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f3aa66d8-3914-45b3-85b3-7b4d64f52c23),volume_id=f3aa66d8-3914-45b3-85b3-7b4d64f52c23,volume_properties=VolumeProperties,volume_type=VolumeType(317c0c8c-2617-43fc-9040-c9d6593a3d29))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:15.722671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1185c0cc-4deb-48f5-8252-a6deff9f72ac) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:15.731499 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:01:15.820325 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 15, 593469)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:15.824500 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:15.826095 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:15.826518 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f3aa66d8-3914-45b3-85b3-7b4d64f52c23) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:15.830433 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:15.593469 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:01:15.831574 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:15.833234 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:01:15.834804 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:01:15.839305 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:15.593469] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:15.844208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:15.845011 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:01:15.845011 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:15.833861 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:15.946974 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1185c0cc-4deb-48f5-8252-a6deff9f72ac) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:15.953222 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-acdf316c-a9be-40c2-913f-af6f25b5bc0d tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (166e0e14-efe7-4e34-9d0d-56586f432bee) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:17.139650 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-52a6e604-3a3c-4fc5-8797-ef9547626682 tempest-AttachVolumeTestJSON-496084914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:18.929857 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-49d06c05-646d-4694-8a69-47f9b041b376 tempest-GroupsTest-1997969427 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:19.547082 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7dc033f6-6a49-46d6-a43c-da6f9cb3bb82 tempest-GroupsTest-1997969427 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:20.980699 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 19, 541031)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:20.981328 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:20.981525 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:20.986492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:20.986925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:20.988089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:20.988089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:19.541031] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:01:20.988089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:20.988515 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:21.012544 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 19, 541031)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:21.013215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:21.018572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:21.019145 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:21.019636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:21.020148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:21.020671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:19.541031] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:01:21.021228 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:21.021747 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:21.022270 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6a52d5d2-ce42-4f72-b37b-cfb19538434b tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:01:22.682932 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (fb6a6a59-ced4-4570-9c2f-389b5cb0fa6c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:22.689829 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (70999542-a25f-4a95-b227-11c19783717c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:22.694985 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (70999542-a25f-4a95-b227-11c19783717c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=2402668a-35a2-41b1-969f-835effd4afa8,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d404bf6c-9ef0-4376-bc54-fe7e2d8f9b55),volume_id=d404bf6c-9ef0-4376-bc54-fe7e2d8f9b55,volume_properties=VolumeProperties,volume_type=VolumeType(e32cea9d-ebc3-4a41-a896-cb7d1fee1dbf))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:22.700723 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0a90fe3a-c486-4680-a7f6-567b1f2c464e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:22.710432 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:01:22.755476 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 19, 541031)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:22.756318 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:22.756871 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:22.758302 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d404bf6c-9ef0-4376-bc54-fe7e2d8f9b55) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:22.760175 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:19.541031 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:22.762189 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:01:22.764108 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:01:22.766434 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:22.768794 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:22.780266 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:22.780991 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:22.782424 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:19.541031] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:22.783242 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:22.783833 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:22.784779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:01:22.788987 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:22.785013 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:22.849880 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0a90fe3a-c486-4680-a7f6-567b1f2c464e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:22.857863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-d1a6ddcf-98a1-459a-afce-8ffbea3ce587 tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (fb6a6a59-ced4-4570-9c2f-389b5cb0fa6c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:24.410880 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (6d3de735-a5b2-42f0-9932-42cd1e1a1677) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:24.417421 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (01cadccf-c7dd-42e3-ad77-522f517002f2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:24.428462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (01cadccf-c7dd-42e3-ad77-522f517002f2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=2402668a-35a2-41b1-969f-835effd4afa8,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(87ebbf12-968b-4e2d-9439-793d2a609cf8),volume_id=87ebbf12-968b-4e2d-9439-793d2a609cf8,volume_properties=VolumeProperties,volume_type=VolumeType(e32cea9d-ebc3-4a41-a896-cb7d1fee1dbf))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:24.463249 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1006e622-3fc6-447d-9662-10f9f9155643) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:24.479286 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:01:24.515728 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 19, 541031)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:24.518364 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:24.519993 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:24.520900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (87ebbf12-968b-4e2d-9439-793d2a609cf8) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:24.525963 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:22.785013 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:24.529702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:24.530123 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:24.530540 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:22.785013] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:24.531033 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:24.531402 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:24.535670 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:01:24.537333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:24.535720 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:24.605970 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1006e622-3fc6-447d-9662-10f9f9155643) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:24.612018 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f15ad1e1-4cef-4b87-b33f-074bbcb0d37e tempest-GroupsTest-1997969427 None] Flow 'volume_create_scheduler' (6d3de735-a5b2-42f0-9932-42cd1e1a1677) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:30.581335 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9daba1da-888f-43f2-8ba0-8759a4dea2dc tempest-GroupsTest-1997969427 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:44.947873 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Flow 'volume_create_scheduler' (5bd95aaa-cd23-4373-914d-1276ed6d55bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:44.962130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (45934ee0-0c33-41b0-bbe2-238c71edc0b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:44.964895 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (45934ee0-0c33-41b0-bbe2-238c71edc0b3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(25698830-0c8e-4474-8c07-bed574669327),volume_id=25698830-0c8e-4474-8c07-bed574669327,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:44.983052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec2c7e85-cfb5-4f6f-a997-3fd21a54dc7f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:44.992799 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:01:45.035539 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 30, 576355)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:45.036205 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:45.036750 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:45.039764 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (25698830-0c8e-4474-8c07-bed574669327) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:45.041427 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:30.576355 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:01:45.043613 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:01:45.045698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:45.047456 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:01:45.050591 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:01:45.052555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:45.052555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:45.052555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:30.576355] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:45.052555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:45.052831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:45.053271 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:01:45.055031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:45.054645 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:45.120568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ec2c7e85-cfb5-4f6f-a997-3fd21a54dc7f) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:45.123532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-91ddddc0-a2bf-4868-ab55-4d9c08f2b54a tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Flow 'volume_create_scheduler' (5bd95aaa-cd23-4373-914d-1276ed6d55bd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:45.840231 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-db3bbfc8-c403-425a-9b72-17c909534505 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:01:46.409858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 30, 576355)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:46.410519 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:46.412904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:46.413453 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:46.414749 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:45.054645 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:46.416324 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:46.417761 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:46.418918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:45.054645] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:46.423373 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:46.423928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:46.424630 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb29b50a-b2ed-446e-891c-8c45f1a9e902 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:46.424258 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:51.582340 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8328c3af-1c41-44f1-b888-f0f6c9287588 tempest-VolumeSnapshotQuotasNegativeTestJSON-1961858864 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:55.219458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-05e9fdba-c7f9-450f-b3bc-f996254107d8 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:01:57.996221 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Flow 'volume_create_scheduler' (ce746fb9-5872-436c-91e5-1a45bac9568d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:58.002858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c7962cea-6799-4ef1-920a-5162c161d8ce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:58.005388 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c7962cea-6799-4ef1-920a-5162c161d8ce) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f014d858-0dfb-4678-ba61-f58553035d8f),volume_id=f014d858-0dfb-4678-ba61-f58553035d8f,volume_properties=VolumeProperties,volume_type=VolumeType(63081374-7c2f-4105-bb57-08c7a66fd118))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:58.007753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (89745d18-a32c-4e70-bcbc-6fd25f118221) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:01:58.016235 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:01:58.035560 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 55, 214270)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:58.036705 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:58.040751 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:58.041273 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f014d858-0dfb-4678-ba61-f58553035d8f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:01:58.041842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:55.214270 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:01:58.043228 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:58.044765 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:01:58.046358 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:01:58.048312 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:01:58.049854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:58.049854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:58.049854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:55.214270] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:58.050064 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:58.050678 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:58.051382 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:01:58.052042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:58.051570 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:01:58.093014 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (89745d18-a32c-4e70-bcbc-6fd25f118221) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:01:58.095675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8ab88749-f74d-4b73-8460-52dc2ba835e6 tempest-VolumeTypesTest-1140104599 None] Flow 'volume_create_scheduler' (ce746fb9-5872-436c-91e5-1a45bac9568d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:01:59.476651 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 1, 55, 214270)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:01:59.481361 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:01:59.482200 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:59.487950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:59.489001 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:01:59.490146 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:01:58.051570] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:01:59.490851 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:01:59.494374 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:01:59.777661 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-732fc222-17cf-42c2-ad18-6ee11ed223ce tempest-VolumeTypesTest-1140104599 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:03.004509 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7ab6d3be-79c5-4a49-9a54-7c6540c26659 tempest-VolumeTypesTest-1140104599 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:18.563239 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Flow 'volume_create_scheduler' (7186a5cf-e52d-4fb1-ac19-dea2ff6e247c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:18.584922 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6f1d4fb6-893e-40a0-a4c5-3c3110efb2cf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:18.589564 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6f1d4fb6-893e-40a0-a4c5-3c3110efb2cf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8be8b2a0-dcff-46c2-93a3-a620aea00b8c),volume_id=8be8b2a0-dcff-46c2-93a3-a620aea00b8c,volume_properties=VolumeProperties,volume_type=VolumeType(03eac2e0-25c9-4d72-95b0-358404e459e3))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:18.598901 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cced2b72-7725-46cc-8e8f-8454a1289bed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:18.609416 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:18.636495 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 2, 999322)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:18.644935 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:18.645318 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:18.645692 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (8be8b2a0-dcff-46c2-93a3-a620aea00b8c) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:18.646507 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:02.999322 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:18.647765 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:18.649698 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:18.651204 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:18.654453 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:02.999322] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:18.662675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:18.663771 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:18.663771 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:18.663771 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:18.651255 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:18.697545 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cced2b72-7725-46cc-8e8f-8454a1289bed) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:18.704481 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e2dce60-1bee-4975-8743-e5e3efb49150 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Flow 'volume_create_scheduler' (7186a5cf-e52d-4fb1-ac19-dea2ff6e247c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:19.971924 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 2, 999322)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:19.972498 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:19.972695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:19.973031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:19.973390 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:19.973766 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:18.651255] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:19.974654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:19.974654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:20.298904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-44f5b1cd-02bb-4705-9838-a0fc11521ae8 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:22.136866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-50cc9b91-e52e-4284-a031-4d6265bc6347 tempest-VolumeRetypeWithoutMigrationTest-1891383966 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:27.364429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (5c533ca3-bd45-4dba-9449-8851b4e8ed48) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:27.366979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c5b3ec70-32fb-45ac-adc9-c2d7245272af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:27.374260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c5b3ec70-32fb-45ac-adc9-c2d7245272af) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c22fef7f-8239-477a-b82d-59ecc34c91bf),volume_id=c22fef7f-8239-477a-b82d-59ecc34c91bf,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:27.376267 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1d6b0d50-8a34-4efb-a92d-407f17a44a8b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:27.399071 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:27.416333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 22, 128036)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:27.416923 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:27.417353 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:27.417778 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c22fef7f-8239-477a-b82d-59ecc34c91bf) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:27.418246 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:22.128036 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:27.419555 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:27.422046 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:27.423943 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:27.426201 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:27.428034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:27.428034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:27.428034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.85,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:22.128036] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:27.428034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:27.428034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:27.428458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:27.429580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:27.427851 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:27.460815 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (1d6b0d50-8a34-4efb-a92d-407f17a44a8b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:27.463605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6a569263-6342-4a6d-90c7-83540f6aaa9d tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (5c533ca3-bd45-4dba-9449-8851b4e8ed48) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:36.204427 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (4c98e92e-089f-477d-b641-16913d6eeb51) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:36.215243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0c04ad5d-17da-4b64-9869-e730c3e84a2d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:36.232982 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0c04ad5d-17da-4b64-9869-e730c3e84a2d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b476a3b3-71ba-4fb2-becb-5c80cf6feb15),volume_id=b476a3b3-71ba-4fb2-becb-5c80cf6feb15,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:36.236660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (09886c05-ea92-4783-be08-19b301b68159) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:36.256150 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:36.278369 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 22, 128036)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:36.279141 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:36.280715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:36.281342 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b476a3b3-71ba-4fb2-becb-5c80cf6feb15) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:36.282000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:27.427851 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:36.283699 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:36.284349 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:36.285505 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:27.427851] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:36.286366 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:36.286957 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:36.287654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:36.288312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:36.287905 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:36.360821 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (09886c05-ea92-4783-be08-19b301b68159) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:36.366149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4acfbf82-8b6b-4847-b791-a215f265d169 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (4c98e92e-089f-477d-b641-16913d6eeb51) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:37.041291 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Flow 'volume_create_scheduler' (69e48e36-5855-41e9-9b32-ceee47c4884a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:37.050913 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f598dc23-577c-474d-b5ff-4726edc5b5d9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:37.053492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f598dc23-577c-474d-b5ff-4726edc5b5d9) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(f4e91c29-493e-416d-a5d0-ccf25a2371f2),volume_id=f4e91c29-493e-416d-a5d0-ccf25a2371f2,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:37.057829 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a2539a39-6db1-4530-a0d9-8317a18914e2) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:37.075346 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:37.108695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 22, 128036)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:37.109097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:37.109467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:37.109838 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (f4e91c29-493e-416d-a5d0-ccf25a2371f2) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:37.110243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:36.287905 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:37.110792 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:37.112494 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:37.114709 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:37.117479 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:37.125437 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:37.125437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:37.125437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:37.125437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:36.287905] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:37.125437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:37.126287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:37.126287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:37.126287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.23, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:37.113659 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:37.160141 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (a2539a39-6db1-4530-a0d9-8317a18914e2) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:37.172080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-dfcdbb43-d9d7-4243-985e-6974988f3b22 tempest-VolumesNegativeTest-2043115879 None] Flow 'volume_create_scheduler' (69e48e36-5855-41e9-9b32-ceee47c4884a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:38.507811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a96efdab-a50c-4e1e-9c0a-f431e818b74f tempest-VolumesActionsTest-1603680914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:39.104459 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (40237385-de51-4a3a-9d32-0a4b29d8bfed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:39.107463 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20d15f61-90be-4176-a1d3-d10dd0426c31) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:39.110124 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (20d15f61-90be-4176-a1d3-d10dd0426c31) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(34dd525a-eacd-4088-a0c3-4100fb3bda3f),volume_id=34dd525a-eacd-4088-a0c3-4100fb3bda3f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:39.112149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7d6ebb54-0db6-4a31-8be4-2c0ccdb3c04b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:39.122765 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:39.141166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 38, 493481)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:39.141496 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:39.142056 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:39.143132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (34dd525a-eacd-4088-a0c3-4100fb3bda3f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:39.143694 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:38.493481 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:39.145310 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:39.147518 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 344.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:39.149229 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:39.151253 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:39.154059 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:39.154059 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:39.158577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:39.158577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:39.158577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:38.493481] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:39.158577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:39.158577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:39.159043 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:39.159765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:39.157826 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:39.198133 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7d6ebb54-0db6-4a31-8be4-2c0ccdb3c04b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:39.200411 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8270b307-197a-4426-a5a6-51c70b2069e4 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (40237385-de51-4a3a-9d32-0a4b29d8bfed) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:41.117235 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4be440c3-26da-43f2-ac0e-860496819d07 tempest-VolumesActionsTest-1603680914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:41.775315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (68baf1c2-2712-4d4d-bcb8-ea33eaa08935) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:41.777821 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9e7960d3-a31f-4c9b-993c-a4f833dbcf86) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:41.780854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9e7960d3-a31f-4c9b-993c-a4f833dbcf86) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a22a136e-40e4-48dc-8fc1-c5bb0da00d08),volume_id=a22a136e-40e4-48dc-8fc1-c5bb0da00d08,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:41.784007 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2c6fd192-a9a8-43ee-a0c7-af8848e2e108) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:41.793844 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:41.814546 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 41, 112743)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:41.815226 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:41.815746 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:41.816263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a22a136e-40e4-48dc-8fc1-c5bb0da00d08) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:41.816866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:41.112743 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:41.818810 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:41.820423 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:41.822137 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:41.826738 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:41.832418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:41.832418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:41.832418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:41.112743] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:41.832418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:41.832418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:41.833241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:41.833241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:41.827107 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:41.877305 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2c6fd192-a9a8-43ee-a0c7-af8848e2e108) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:41.880075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-31f1ece1-cb3a-422e-ae98-7a4d2de8f095 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (68baf1c2-2712-4d4d-bcb8-ea33eaa08935) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:44.122605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fbdfaef2-c1b2-4b31-9e16-335cd11ec409 tempest-VolumesActionsTest-1603680914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:44.939376 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (6728f9fa-8e6b-4c80-80d3-168767e09eaf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:44.942614 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ba0aace8-d288-47c9-bf2d-f375accd9169) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:44.968315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ba0aace8-d288-47c9-bf2d-f375accd9169) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0a30dd92-59b2-4c48-9bd6-e21b0e3f310e),volume_id=0a30dd92-59b2-4c48-9bd6-e21b0e3f310e,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:44.973728 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f871dfe7-f997-43d4-b78c-a24da559c57c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:45.034240 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:45.065555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 44, 115820)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:45.070742 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:45.071758 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:45.072289 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0a30dd92-59b2-4c48-9bd6-e21b0e3f310e) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:45.072987 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:44.115820 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:45.075092 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:45.075640 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:45.076161 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:44.115820] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:45.076881 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:45.077384 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:45.078011 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:45.078563 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:45.078189 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:45.118056 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f871dfe7-f997-43d4-b78c-a24da559c57c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:45.120426 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c2cf0f96-c17b-4b7c-9d06-a4cb874266a7 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (6728f9fa-8e6b-4c80-80d3-168767e09eaf) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:45.846900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-04783d76-919e-42b2-bb76-8ac5ac786841 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:02:47.348578 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-60d80e88-b9d0-40b0-a285-b6d32dde1e4d tempest-VolumesActionsTest-1603680914 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:47.850261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (eb98bcd3-7770-4f70-99e6-9e7324feb4cc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:47.853810 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4cb9ad71-ee74-4cbb-9ae8-44b8819b4e75) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:47.856869 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4cb9ad71-ee74-4cbb-9ae8-44b8819b4e75) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(759a144e-87c7-482b-9437-c4ecac055f9b),volume_id=759a144e-87c7-482b-9437-c4ecac055f9b,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:47.860098 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3e042262-9e39-4dcc-b96c-d6f9fe405379) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:47.868668 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:47.889628 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 47, 343361)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:47.890960 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:47.891337 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:47.891862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (759a144e-87c7-482b-9437-c4ecac055f9b) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:47.893596 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:47.343361 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:47.894085 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:47.896240 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:47.898973 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:47.901145 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:47.904644 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:47.904644 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:47.904644 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:47.343361] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:47.904644 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:47.904644 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:47.905075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:47.905075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:47.897101 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:47.950531 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3e042262-9e39-4dcc-b96c-d6f9fe405379) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:47.957897 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7ddcc2b2-cbfe-4e37-9ea2-639874ef6ae8 tempest-VolumesActionsTest-2028875331 None] Flow 'volume_create_scheduler' (eb98bcd3-7770-4f70-99e6-9e7324feb4cc) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:50.242967 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a0ff629e-50e1-4763-8010-70a01dd7b6c8 tempest-VolumesActionsTest-2028875331 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:51.726918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1b6ef20a-1b44-49fa-9f35-458022993ef8 tempest-VolumesActionsTest-2028875331 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:55.211474 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-55f98432-5667-4084-ba9c-6d03fab53f59 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:02:59.663418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Flow 'volume_create_scheduler' (5a707f53-ff42-4acf-99b9-93d655cdc863) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:02:59.669327 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (692bcb34-c2bf-454b-a49e-754579ac6556) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:59.680115 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (692bcb34-c2bf-454b-a49e-754579ac6556) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c1fb6e0c-6beb-4960-ae37-f401a74076fb),volume_id=c1fb6e0c-6beb-4960-ae37-f401a74076fb,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:59.683175 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fd087e4e-b7fc-4f17-b94c-706a85a20dc9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:02:59.690711 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:02:59.705967 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 55, 204611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:02:59.706713 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:02:59.707223 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:59.707753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c1fb6e0c-6beb-4960-ae37-f401a74076fb) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:02:59.708309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:55.204611 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:59.709539 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:02:59.711996 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:02:59.713861 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:02:59.718498 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:02:59.724637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:59.724637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:02:59.724637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:55.204611] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:02:59.724637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:02:59.724637 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:02:59.726677 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:02:59.726677 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:59.711982 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:02:59.754201 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fd087e4e-b7fc-4f17-b94c-706a85a20dc9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:02:59.757373 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2108ee07-a661-4f9d-87f1-9dbee199f748 tempest-SnapshotMetadataTestJSON-1506160061 None] Flow 'volume_create_scheduler' (5a707f53-ff42-4acf-99b9-93d655cdc863) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:01.100505 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 2, 55, 204611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:03:01.102078 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:03:01.102709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:01.103344 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:03:01.104031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:59.711982 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:03:01.105461 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:01.106002 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:01.106646 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:02:59.711982] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:03:01.106931 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:03:01.107892 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:03:01.108283 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bc1c0bc8-54ae-4dbb-9c6a-93c43f86fc9a tempest-SnapshotMetadataTestJSON-1506160061 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:01.107938 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:03:05.265662 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-59cb65a7-4cbd-422f-bf14-ca4292c17683 tempest-VolumesNegativeTest-2043115879 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:03:05.380336 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Flow 'volume_create_scheduler' (9ea8559e-faa4-4b69-b8d9-d0814b4c1a03) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:05.391875 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a5b83bc8-1d0e-4dd7-8891-fde246d26778) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:05.397591 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a5b83bc8-1d0e-4dd7-8891-fde246d26778) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c7910b47-de22-4de6-898a-baf89e2af426),volume_id=c7910b47-de22-4de6-898a-baf89e2af426,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:05.399587 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7cff0bdd-1665-46ea-af84-c526afbe6fdf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:05.417132 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:03:05.472188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 5, 253608)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:03:05.472705 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:03:05.473072 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:05.473632 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c7910b47-de22-4de6-898a-baf89e2af426) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:03:05.474194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:05.253608 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:03:05.477332 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:03:05.479177 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:03:05.481011 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:05.483694 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:05.485960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:03:05.485960 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:03:05.486835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:05.487400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:05.487966 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:05.253608] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:03:05.488726 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:03:05.489228 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:03:05.489863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:03:05.490409 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:05.490043 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:03:05.530766 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7cff0bdd-1665-46ea-af84-c526afbe6fdf) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:05.539603 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-37b8356c-e7a6-447b-abb0-d24f2fa41d35 tempest-DeleteServersTestJSON-1962391540 None] Flow 'volume_create_scheduler' (9ea8559e-faa4-4b69-b8d9-d0814b4c1a03) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:06.564990 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-457985ff-b2fc-4f1e-bca6-7361d18e01a5 tempest-SnapshotMetadataTestJSON-1506160061 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:03:13.815324 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Flow 'volume_create_scheduler' (1a337d2a-f35a-472d-90f1-60ded6e5d1a0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:13.819964 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (90c5b441-1d4a-416a-82e0-f37fa1723f5f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:13.823945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (90c5b441-1d4a-416a-82e0-f37fa1723f5f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d7c48b2a-ad5c-4686-966c-146b3eefe548),volume_id=d7c48b2a-ad5c-4686-966c-146b3eefe548,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:13.826722 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (130c8dc5-ec06-49e9-b757-b4e021bef09b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:13.835935 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:03:13.863253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 6, 559205)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:03:13.863729 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:03:13.864199 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:13.864767 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d7c48b2a-ad5c-4686-966c-146b3eefe548) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:03:13.865527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:06.559205 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:03:13.873113 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:03:13.874887 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:03:13.879811 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:03:13.888006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:03:13.895416 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:03:13.895416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:13.895416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:13.895416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:06.559205] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:03:13.895416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:03:13.896096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:03:13.896096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:03:13.896096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:13.875910 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:03:13.916622 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (130c8dc5-ec06-49e9-b757-b4e021bef09b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:13.921151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-49f635fb-6599-407f-a674-c10dff10efc1 tempest-AbsoluteLimitsTests-1203552874 None] Flow 'volume_create_scheduler' (1a337d2a-f35a-472d-90f1-60ded6e5d1a0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:16.187976 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-172e1f29-8e3a-45ca-bb4b-b1399fdc8cac tempest-AbsoluteLimitsTests-1203552874 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:03:45.848441 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4660a4a6-c6a7-4deb-9bd1-92d594e1ed56 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:03:54.349096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Flow 'volume_create_scheduler' (242ba252-f4b5-40be-994c-83f73a402d24) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:54.359215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0323e49f-563a-4570-aab1-d23d1430978e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:54.365594 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0323e49f-563a-4570-aab1-d23d1430978e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7b7fe2cc-e7c8-4e38-a28e-c31004f8fe0d),volume_id=7b7fe2cc-e7c8-4e38-a28e-c31004f8fe0d,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:54.368407 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8ebaf8c6-86d4-4312-8bbb-8afe65d18823) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:03:54.378518 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:03:54.432159 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 16, 182911)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:03:54.432888 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:03:54.434182 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:54.434843 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (7b7fe2cc-e7c8-4e38-a28e-c31004f8fe0d) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:03:54.435249 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:16.182911 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:03:54.441002 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:03:54.442881 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:03:54.445672 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:03:54.448917 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:03:54.453555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:54.453555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:03:54.453555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:16.182911] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:03:54.453555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Weigher CapacityWeigher returned, weigher value is {max: 386.0, min: 386.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:03:54.453555 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:03:54.456429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:03:54.456429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:54.443533 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:03:54.549664 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8ebaf8c6-86d4-4312-8bbb-8afe65d18823) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:03:54.553424 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-cb5f1f90-78d5-4f1d-b975-bcafa11963ef tempest-TestVolumeBackupRestore-1162231384 None] Flow 'volume_create_scheduler' (242ba252-f4b5-40be-994c-83f73a402d24) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:03:56.053218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f7667c2c-cf73-46b8-8bc2-2b2a363a3361 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:03:57.086579 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-c7fbea7c-0edb-4326-b847-f101d51707a3 tempest-TestVolumeBackupRestore-1162231384 None] create_backup in rpcapi backup_id 750f3b0c-4514-4200-82f6-5f96dd472cdc {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:04:15.344292 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Flow 'volume_create_scheduler' (3758368e-e275-4576-a9e3-f3bec3fa2fc1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:15.348204 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (32af08b0-39e0-4157-a5b3-356034913f20) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:15.357276 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (32af08b0-39e0-4157-a5b3-356034913f20) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1db74c9b-76be-4978-a8ac-3b6cb127fc26),volume_id=1db74c9b-76be-4978-a8ac-3b6cb127fc26,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:15.362322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5d0133d0-2323-4b9e-87f8-0b1fa44670a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:15.372163 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:04:15.392814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:15.393895 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:15.394926 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:15.395461 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1db74c9b-76be-4978-a8ac-3b6cb127fc26) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:15.396025 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:56.047552 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:15.397783 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:15.398347 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:15.398921 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.22, total_capacity_gb: 19.85,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:03:56.047552] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:15.399595 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Weigher CapacityWeigher returned, weigher value is {max: 385.0, min: 385.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:15.400266 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:15.400916 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:04:15.401522 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:15.401113 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:15.439116 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5d0133d0-2323-4b9e-87f8-0b1fa44670a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:15.441688 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b73ee75c-e34a-411b-844d-f5c0cf3df303 tempest-TestVolumeBackupRestore-1162231384 None] Flow 'volume_create_scheduler' (3758368e-e275-4576-a9e3-f3bec3fa2fc1) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:18.554102 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Flow 'volume_create_scheduler' (4306f056-32e1-46d8-a26f-215012db7dde) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:18.559558 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9053a832-893a-461b-a1b6-a3437007b217) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:18.563359 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9053a832-893a-461b-a1b6-a3437007b217) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d4cf6c7f-0e5a-4771-9c00-a456ed9aee8f),volume_id=d4cf6c7f-0e5a-4771-9c00-a456ed9aee8f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:18.570696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (15ae440a-f105-4d97-b71d-fa44d5018061) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:18.582235 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:04:18.663327 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:18.663939 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:18.704708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:18.707226 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d4cf6c7f-0e5a-4771-9c00-a456ed9aee8f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:18.709791 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:15.401113 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:18.711128 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:04:18.713701 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:18.715957 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:18.719365 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:18.727886 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:18.728820 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:18.732055 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.22, total_capacity_gb: 19.85,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:15.401113] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:18.732831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Weigher CapacityWeigher returned, weigher value is {max: 384.0, min: 384.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:18.733426 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:18.737067 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:04:18.737842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.22, total_capacity_gb: 19.85,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:18.737358 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:18.817462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (15ae440a-f105-4d97-b71d-fa44d5018061) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:18.821857 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7bb91cbe-7c2d-4d97-b4ed-ec847ee13cb1 tempest-ServersTestBootFromVolume-308351442 None] Flow 'volume_create_scheduler' (4306f056-32e1-46d8-a26f-215012db7dde) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:29.311638 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Flow 'volume_create_scheduler' (29b7096d-81da-4ce9-a19e-c7c02b5294b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:29.325098 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (87d5f1ca-1bf8-4931-a6db-408944dd7cab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:29.327836 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (87d5f1ca-1bf8-4931-a6db-408944dd7cab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d0a3b3f6-3a92-444c-a624-670dd03859f9),volume_id=d0a3b3f6-3a92-444c-a624-670dd03859f9,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:29.330344 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2921758-7847-407c-bd62-f7692e674461) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:04:29.338821 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:04:29.357066 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:29.357958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:29.358634 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:29.359222 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d0a3b3f6-3a92-444c-a624-670dd03859f9) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:29.359826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.22, total_capacity_gb: 19.85,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:18.737358 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:04:29.361181 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:29.363515 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:29.369121 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:04:29.373480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:04:29.376980 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:04:29.376980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:29.376980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:29.376980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.22, total_capacity_gb: 19.85,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:18.737358] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:29.376980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 383.0, min: 383.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:29.377756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:29.377756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:04:29.377756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.22, total_capacity_gb: 19.85,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:29.371054 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:29.449785 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2921758-7847-407c-bd62-f7692e674461) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:04:29.461640 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c95c7332-2e64-4327-8130-270e2198a50c tempest-SnapshotsActionsTest-1253319703 None] Flow 'volume_create_scheduler' (29b7096d-81da-4ce9-a19e-c7c02b5294b3) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:04:30.642354 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:30.643017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:30.643361 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:30.644132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:30.644611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.22, total_capacity_gb: 19.85,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:29.371054 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:30.648736 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:30.649333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:30.649938 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.22, total_capacity_gb: 19.85,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:29.371054] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:30.651642 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 382.0, min: 382.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:30.652386 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:30.653352 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf49f87b-b5d7-4631-a5d7-44dc4d0da884 tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:30.652652 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:33.456826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:33.465687 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:33.466060 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:33.466424 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:33.466849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:30.652652 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:33.467673 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 344.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:33.469770 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 324.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:33.473787 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:33.477481 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:33.482307 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:33.482307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:33.482307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:33.482307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:30.652652] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:33.482307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:33.483108 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:33.483108 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c62df43-d614-4e62-870c-e41f41867509 tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:33.482093 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:37.390032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:37.391178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:37.391708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:37.392437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:37.393628 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:33.482093 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:37.396235 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:37.397667 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:37.398222 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 17, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 17,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:33.482093] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:37.399220 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 380.0, min: 380.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:37.399798 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:37.400375 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b69e6039-bd44-4c2a-a5ab-bf4e88590ac4 tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:37.400016 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:41.454609 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:41.456596 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:41.460144 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:41.460527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:41.460925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:37.400016 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:41.462095 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:41.462457 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:41.462866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 18, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 18,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:37.400016] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:41.464017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 379.0, min: 379.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:41.478001 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:41.478600 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-db1efda6-6124-477e-9f75-2910784c6af3 tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:41.477996 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:45.407781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.85, 'free_capacity_gb': 19.22, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 3, 56, 47552)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:04:45.408491 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:04:45.409593 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:45.410947 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:04:45.411469 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:41.477996 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 304.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:45.412351 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:45.415800 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 284.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:04:45.418278 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 264.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:04:45.421923 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 19, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 19,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:41.477996] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:04:45.425953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Weigher CapacityWeigher returned, weigher value is {max: 378.0, min: 378.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:04:45.427454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:04:45.427454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e31643-50d8-4ca2-b983-c24750814498 tempest-SnapshotsActionsTest-1253319703 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 11.219999999999999, total_capacity_gb: 19.85,allocated_capacity_gb: 20, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 20,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:45.415039 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:04:45.850413 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d3800da5-19b8-4efe-a058-1bfb746ac950 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:04:52.042370 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7861b324-27f3-4131-babb-26f2be2325cf tempest-SnapshotsActionsTest-1253319703 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:04:55.964609 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f613e038-e3d0-4265-8970-98e3fe0c8709 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:04.135784 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Flow 'volume_create_scheduler' (e21ab0a8-f687-40a5-820d-33a646976751) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:04.142835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7b65b9a1-960c-4d96-a3eb-86fdd184fa14) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:04.146133 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7b65b9a1-960c-4d96-a3eb-86fdd184fa14) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(eb81d485-b67a-406b-85b9-78857b64e343),volume_id=eb81d485-b67a-406b-85b9-78857b64e343,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:04.154587 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (594cb3be-59e2-4ed7-aa07-5760aaa14cf9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:04.168898 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:05:04.185447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 4, 55, 955076)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:05:04.186345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:05:04.186962 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:04.187586 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (eb81d485-b67a-406b-85b9-78857b64e343) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:05:04.188360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.39,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:55.955076 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:04.189805 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 244.39999999999998, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:04.191826 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:05:04.193971 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:04.195847 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:04.200151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:04.201048 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:04.202283 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.39,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:04:55.955076] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:05:04.203096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Weigher CapacityWeigher returned, weigher value is {max: 373.8, min: 373.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:05:04.203509 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:05:04.203941 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:05:04.204544 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.39,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:04.204175 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:05:04.245615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (594cb3be-59e2-4ed7-aa07-5760aaa14cf9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:04.249319 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d5e3d29-102f-4801-bcf8-cecb011cc74d tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Flow 'volume_create_scheduler' (e21ab0a8-f687-40a5-820d-33a646976751) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:07.860713 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a698bc60-1cdc-40be-8cfe-d96a8c7e34ff tempest-VolumeQuotasNegativeTestJSON-1158831787 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:13.333925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (906224d1-86bb-4747-9f5a-3f3710de8e55) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:13.349212 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6a61e181-e498-4c37-8399-82d937c3a06b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:13.359605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6a61e181-e498-4c37-8399-82d937c3a06b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(eadf804a-f17e-4ea4-b992-c925e6e57a8e),volume_id=eadf804a-f17e-4ea4-b992-c925e6e57a8e,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:13.369711 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (28115abc-bc5e-455e-89b5-fdbec8ee1c5c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:13.394360 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:05:13.426824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 7, 851664)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:05:13.429880 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:05:13.430916 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:13.433164 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (eadf804a-f17e-4ea4-b992-c925e6e57a8e) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:05:13.434559 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.39,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:07.851664 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:05:13.437105 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:13.437979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:13.445083 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.39,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:07.851664] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:05:13.447971 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Weigher CapacityWeigher returned, weigher value is {max: 373.8, min: 373.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:05:13.448510 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:05:13.450721 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:05:13.452335 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.39,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:13.451924 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:05:13.504072 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (28115abc-bc5e-455e-89b5-fdbec8ee1c5c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:13.506265 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ccea1964-270c-4327-9b21-6c481c888a2d tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (906224d1-86bb-4747-9f5a-3f3710de8e55) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:14.728837 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-42b6b6d5-3e80-4ea5-bb32-bab57e9993db tempest-VolumesBackupsAdminTest-953270115 None] create_backup in rpcapi backup_id ad504e7d-6609-46da-9e6d-d50578f9abe2 {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:05:22.246912 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (6341a805-2cbc-42eb-968a-f9fb29edd9af) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:22.250331 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (414691fc-ad64-4e33-a696-c084747b93b3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:22.253674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (414691fc-ad64-4e33-a696-c084747b93b3) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b7db60e7-b0d8-41c9-9119-7101f0a2232d),volume_id=b7db60e7-b0d8-41c9-9119-7101f0a2232d,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:22.256309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0ef478cf-e89c-41b8-8b70-d81cbd111071) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:22.266743 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:05:22.284741 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 7, 851664)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:05:22.285272 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:05:22.285695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:22.286281 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b7db60e7-b0d8-41c9-9119-7101f0a2232d) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:05:22.286917 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.39,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:13.451924 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:22.288233 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:05:22.292646 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:22.294861 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:22.296807 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:22.300410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:22.300410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:22.300410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.39,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:13.451924] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:05:22.300410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Weigher CapacityWeigher returned, weigher value is {max: 372.8, min: 372.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:05:22.300410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:05:22.301121 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:05:22.301121 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.39,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:22.297698 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:05:22.329351 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0ef478cf-e89c-41b8-8b70-d81cbd111071) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:22.331490 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15f6d006-8fa7-412a-a50c-40ad11c3414c tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (6341a805-2cbc-42eb-968a-f9fb29edd9af) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:23.002865 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6c31ceba-1562-406b-9347-8c6ce0e28290 tempest-TaggedBootDevicesTest-262552579 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:24.268597 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-70221c30-a635-40f3-8488-660a44b1e44d tempest-TaggedBootDevicesTest-262552579 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:26.189318 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c7e6b0a3-d4f8-49f6-b869-37ddc0af68f4 tempest-TaggedBootDevicesTest-262552579 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.39, 'free_capacity_gb': 18.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:28.819311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8a3be220-1520-40bf-b3bc-ae7eb85436ce tempest-VolumesBackupsAdminTest-953270115 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.38, 'free_capacity_gb': 18.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:30.688121 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (ace93aec-297b-4eca-a0ca-23012ca5bf60) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:30.693864 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b8479305-bbb3-4a44-b938-2a8241b7f855) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:30.705258 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b8479305-bbb3-4a44-b938-2a8241b7f855) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a3673216-b0df-4687-99ed-25ae706253c1),volume_id=a3673216-b0df-4687-99ed-25ae706253c1,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:30.707572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6713d7b3-5361-4d8d-8576-da93a1edaf44) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:30.728335 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:05:30.775514 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.38, 'free_capacity_gb': 18.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 28, 813174)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:05:30.776752 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:05:30.778556 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:30.779415 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a3673216-b0df-4687-99ed-25ae706253c1) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:05:30.781038 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.38,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:28.813174 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:05:30.783207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:30.785593 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 345.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:30.790531 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:05:30.796771 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:05:30.802907 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:05:30.807316 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:30.810973 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:30.811527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.38,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:28.813174] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:05:30.812123 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Weigher CapacityWeigher returned, weigher value is {max: 375.59999999999997, min: 375.59999999999997} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:05:30.814640 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:05:30.815236 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:05:30.816293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.38,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:30.815270 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:05:30.876917 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6713d7b3-5361-4d8d-8576-da93a1edaf44) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:30.882461 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b154c0d7-8051-4099-b4cc-e0191ee1924a tempest-VolumesBackupsAdminTest-953270115 None] Flow 'volume_create_scheduler' (ace93aec-297b-4eca-a0ca-23012ca5bf60) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:32.042285 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-cd2acf9e-c3ca-4e36-bb97-00b244ac2a05 tempest-VolumesBackupsAdminTest-953270115 None] create_backup in rpcapi backup_id 21b5d108-c255-4f05-b5c2-0637b5ea5cb8 {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:05:41.065473 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cf10ab39-68b3-497f-b8ee-160070469779 tempest-VolumesBackupsAdminTest-953270115 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.36, 'free_capacity_gb': 18.29, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:42.928033 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-54d97a70-18f0-4685-a9ce-23c0d6bc9ba5 tempest-VolumesBackupsAdminTest-953270115 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.36, 'free_capacity_gb': 18.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:45.853216 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4bf3611a-5a57-4ae3-9ff9-a3e932b23528 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:05:49.489483 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Flow 'volume_create_scheduler' (f6b7d697-c037-4792-9eb5-f9d18cbbdb0c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:49.493206 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b3867df4-41a9-46f9-b133-4a2c9bd1c420) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:49.502899 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b3867df4-41a9-46f9-b133-4a2c9bd1c420) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b2c77fd4-88b5-43fe-9d17-ef0b80b51639),volume_id=b2c77fd4-88b5-43fe-9d17-ef0b80b51639,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:49.508672 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d54ff174-7f98-44e4-a88f-3242e0fc7100) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:05:49.525792 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:05:49.552409 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.36, 'free_capacity_gb': 18.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 42, 920395)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:05:49.552838 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:05:49.553255 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:49.553627 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b2c77fd4-88b5-43fe-9d17-ef0b80b51639) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:05:49.554054 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.3, total_capacity_gb: 19.36,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:42.920395 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:49.555335 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:05:49.557157 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:05:49.560500 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:05:49.564358 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.3, total_capacity_gb: 19.36,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:42.920395] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:05:49.568814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Weigher CapacityWeigher returned, weigher value is {max: 376.2, min: 376.2} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:05:49.569755 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:05:49.569755 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:05:49.569755 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.3, total_capacity_gb: 19.36,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:49.557612 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:05:49.622965 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d54ff174-7f98-44e4-a88f-3242e0fc7100) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:05:49.625586 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-eb44cda9-892a-485b-bf01-e2f294bb5a02 tempest-VolumesImageMetadata-1125020998 None] Flow 'volume_create_scheduler' (f6b7d697-c037-4792-9eb5-f9d18cbbdb0c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:05:54.451945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5094ae4e-c0f8-4d81-8c31-ad7e4d62ce05 tempest-VolumesImageMetadata-1125020998 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.35, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:05:56.043453 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f4700b84-ef0e-4ecf-9ee9-b4a9ba6cb7e5 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.35, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:00.117015 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Flow 'volume_create_scheduler' (ab1779d8-f1f9-4c85-9a67-9b1db14f22b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:00.125202 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2ce06aab-1bfe-4296-93ab-716239e29be8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:00.134109 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2ce06aab-1bfe-4296-93ab-716239e29be8) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(494c2d6c-632c-423b-9d9f-8326480a0879),volume_id=494c2d6c-632c-423b-9d9f-8326480a0879,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:00.136283 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0a769595-bcd8-4506-b203-57f8a4592d76) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:00.154399 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:00.212262 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.35, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 56, 12577)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:00.215132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:00.216364 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:00.217473 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (494c2d6c-632c-423b-9d9f-8326480a0879) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:00.220605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.35,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:56.012577 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:00.226330 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:00.228581 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:00.230909 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.28, total_capacity_gb: 19.35,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:05:56.012577] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:00.231904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Weigher CapacityWeigher returned, weigher value is {max: 376.0, min: 376.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:00.234413 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:00.234960 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:00.237309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.35,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:00.234992 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:00.362057 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0a769595-bcd8-4506-b203-57f8a4592d76) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:00.369654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-de8b4cd1-ce7f-4ec7-bacf-d6e4707ee6cf tempest-VolumesDeleteCascade-377355327 None] Flow 'volume_create_scheduler' (ab1779d8-f1f9-4c85-9a67-9b1db14f22b8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:01.570173 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.35, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 56, 12577)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:01.570980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:01.570980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:01.572046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:01.572046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.35,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:00.234992 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:01.587877 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:01.590863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:01.593495 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 365.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:01.595845 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:01.600430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:01.600430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:01.600430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.28, total_capacity_gb: 19.35,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:00.234992] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:01.600430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Weigher CapacityWeigher returned, weigher value is {max: 375.0, min: 375.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:01.600430 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:01.601474 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b1b405ac-ae8c-4713-8884-456a610b5fc0 tempest-VolumesDeleteCascade-377355327 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.35,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:01.589953 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:02.869495 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.35, 'free_capacity_gb': 18.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 5, 56, 12577)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:02.870151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:02.870251 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:02.870657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:02.871097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.35,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:01.589953 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:02.872263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:02.872648 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:02.873015 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.28, total_capacity_gb: 19.35,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:01.589953] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:02.873566 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Weigher CapacityWeigher returned, weigher value is {max: 374.0, min: 374.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:02.881270 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:02.881853 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c0a1c52-2260-4291-ab22-84042388e170 tempest-VolumesDeleteCascade-377355327 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.280000000000001, total_capacity_gb: 19.35,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:02.881325 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:07.800218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-807d63f8-da1c-4f98-8a04-3ca7942465de tempest-TestVolumeBackupRestore-1162231384 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.33, 'free_capacity_gb': 18.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:08.601076 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d7bdd255-14d8-4b92-a372-ee736a2b4a70 tempest-TestVolumeBackupRestore-1162231384 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.33, 'free_capacity_gb': 18.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:10.265308 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d9102db8-9498-49d9-8ed7-f56b547752c6 tempest-VolumesDeleteCascade-377355327 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.33, 'free_capacity_gb': 18.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:12.108628 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3340cbc0-de4d-4047-b029-8b19cececb6c tempest-TaggedBootDevicesTest_v242-10492700 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.12, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:13.363149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b7e69c4d-2d6d-42a2-b9fb-ccceae0570ae tempest-TaggedBootDevicesTest_v242-10492700 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.12, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:15.479109 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7bad1d49-238d-4fe8-821e-807e8e52f6d2 tempest-TaggedBootDevicesTest_v242-10492700 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.12, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:15.510459 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Flow 'volume_create_scheduler' (94698f79-4c88-43d8-a33b-78f680baf1df) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:15.518411 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3e27ec2e-336f-4c41-97a8-702542314b58) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:15.524533 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3e27ec2e-336f-4c41-97a8-702542314b58) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(78342019-478e-43a5-8d0b-38bf722c00ec),volume_id=78342019-478e-43a5-8d0b-38bf722c00ec,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:15.531267 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (71609239-7065-48e1-90ff-370d084d7b07) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:15.546091 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:15.572225 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.12, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 15, 467902)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:15.573128 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:15.573928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:15.575134 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (78342019-478e-43a5-8d0b-38bf722c00ec) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:15.575755 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.12, total_capacity_gb: 19.78,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:15.467902 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 345.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:15.577223 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:15.578869 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 325.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:15.580484 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:15.582368 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:15.583846 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:15.583846 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:15.585015 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:15.585015 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.12, total_capacity_gb: 19.78,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:15.467902] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:15.585809 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Weigher CapacityWeigher returned, weigher value is {max: 389.6, min: 389.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:15.586280 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:15.587092 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:15.587592 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.12, total_capacity_gb: 19.78,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:15.587122 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:15.667014 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (71609239-7065-48e1-90ff-370d084d7b07) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:15.669147 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-691e44c3-60be-44e0-abea-b781d0b8236e tempest-VolumesBackupsV39Test-978310112 None] Flow 'volume_create_scheduler' (94698f79-4c88-43d8-a33b-78f680baf1df) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:16.751570 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-c2327310-c652-4ed1-a0ad-776a7d38ce4a tempest-VolumesBackupsV39Test-978310112 None] create_backup in rpcapi backup_id 95feb87e-ef57-46dc-8525-244568fa8eed {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:06:24.760948 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bd42295e-6542-4664-bbd2-983a87f2d9d9 tempest-VolumesBackupsV39Test-978310112 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:27.580871 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9d279cf5-a57a-4d65-93dd-a26bf763a0f2 tempest-DeleteServersTestJSON-1962391540 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:29.325112 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (6333315c-655c-4ce5-80b3-91a485acfda4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:29.332166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b0ceb90-316c-4dfe-ab2a-405576be30b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:29.335418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8b0ceb90-316c-4dfe-ab2a-405576be30b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5cd89177-c7b1-467e-b0fe-cd8b8c3589fd),volume_id=5cd89177-c7b1-467e-b0fe-cd8b8c3589fd,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:29.337723 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3722f049-6540-4117-a47a-5fec1181e43a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:29.354516 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:29.387080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 27, 576441)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:29.390683 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:29.391248 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:29.391800 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5cd89177-c7b1-467e-b0fe-cd8b8c3589fd) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:29.392433 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:27.576441 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:29.393745 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 382.40000000000003, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:29.395566 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:29.400842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:29.403962 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:29.406945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:29.406945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:29.406945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:27.576441] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:29.406945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:29.406945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:29.407404 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:29.407404 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:29.397308 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:29.453626 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (3722f049-6540-4117-a47a-5fec1181e43a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:29.459091 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4292e7e9-249b-4f63-af6f-14b17c451abb tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (6333315c-655c-4ce5-80b3-91a485acfda4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:31.972383 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (c31cee83-e42e-4a69-ad29-53d504cd47b8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:31.979376 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a234122b-65e4-4b8f-aa5f-dc777193a18d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:31.988999 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a234122b-65e4-4b8f-aa5f-dc777193a18d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=5cd89177-c7b1-467e-b0fe-cd8b8c3589fd,volume=Volume(ead0b6e9-032d-4c39-92d2-ce9d5b911a9f),volume_id=ead0b6e9-032d-4c39-92d2-ce9d5b911a9f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:31.995288 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (52f8eb07-bddc-4bb3-9001-64ef6c34be2a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:32.014070 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:32.048116 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 27, 576441)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:32.048631 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:32.049070 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:32.049617 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ead0b6e9-032d-4c39-92d2-ce9d5b911a9f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:32.050313 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:29.397308 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:32.052021 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:32.052485 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:32.053746 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:29.397308] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:32.054391 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Weigher CapacityWeigher returned, weigher value is {max: 389.4, min: 389.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:32.054853 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:32.055377 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:32.055825 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:32.055408 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:32.093551 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (52f8eb07-bddc-4bb3-9001-64ef6c34be2a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:32.099333 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-312a67e3-5833-407e-96ba-9487cee94dab tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (c31cee83-e42e-4a69-ad29-53d504cd47b8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:34.561039 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (b018feeb-18a7-43be-b1b8-a9418089e555) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:34.564719 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ed11a963-f88e-4c21-a16e-568398db4877) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:34.568164 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ed11a963-f88e-4c21-a16e-568398db4877) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e5394b85-abbe-4699-a168-0b4af15fd481),volume_id=e5394b85-abbe-4699-a168-0b4af15fd481,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:34.571200 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e8dab91e-3b7b-456c-80f4-b710d5a00e68) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:34.579749 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:34.617928 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 27, 576441)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:34.619342 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:34.619920 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:34.620445 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e5394b85-abbe-4699-a168-0b4af15fd481) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:34.620863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:32.055408 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:34.621925 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:34.623815 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:34.625614 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:34.627707 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:34.630296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:34.630296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:34.630296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.23, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:32.055408] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:34.630296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:34.630296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:34.632393 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:34.632393 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.23, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:34.624537 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:34.661371 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e8dab91e-3b7b-456c-80f4-b710d5a00e68) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:34.664180 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9201791c-a45a-4066-90a4-187b58e39fd3 tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (b018feeb-18a7-43be-b1b8-a9418089e555) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:36.142094 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (c365acef-1640-4f02-8c5a-75918caab495) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:36.146845 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a975fb21-3cf9-4ad8-a661-8e94dd6c07dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:36.156863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a975fb21-3cf9-4ad8-a661-8e94dd6c07dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=e5394b85-abbe-4699-a168-0b4af15fd481,volume=Volume(86d22aa0-2169-459d-8ed7-d13b80b9d5be),volume_id=86d22aa0-2169-459d-8ed7-d13b80b9d5be,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:36.161831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2713edcb-7189-4e1c-829d-1ae9cfb6f6f3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:36.176979 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:36.197500 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 27, 576441)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:36.198197 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:36.198831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:36.199366 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (86d22aa0-2169-459d-8ed7-d13b80b9d5be) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:36.199939 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.23, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:34.624537 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:36.201364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:36.202863 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 344.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:36.204207 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:36.205684 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:36.207705 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:36.208699 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:36.209418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:36.209945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.23, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:34.624537] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:36.210375 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:36.210767 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:36.211132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:36.211517 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.23, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:36.211161 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:36.247863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2713edcb-7189-4e1c-829d-1ae9cfb6f6f3) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:36.250727 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-e2e91f14-4b77-412b-855a-d405dc9e0622 tempest-VolumesCloneTest-162905673 None] Flow 'volume_create_scheduler' (c365acef-1640-4f02-8c5a-75918caab495) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:39.953279 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e3dc9a73-f744-4dc5-8248-3477d654d84a tempest-VolumesCloneTest-162905673 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:41.659311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-28a49fd2-834c-4697-b2ae-bbd672c95ff7 tempest-VolumesCloneTest-162905673 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:43.511088 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a21c3641-0cca-411f-b742-3b767305c41f tempest-VolumesCloneTest-162905673 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:45.845097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-381d1c28-a346-410b-ab36-3de44ada56db None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:06:46.044128 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf4e22b3-8a31-4b91-a3bb-796d494bcf2e tempest-VolumesCloneTest-162905673 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:48.608345 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Flow 'volume_create_scheduler' (570c1b9a-8e89-4862-b7d9-0c8cc7634cf9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:48.610817 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7f757ac4-b364-4c98-9d6a-873dc1dc3e7b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:48.613727 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7f757ac4-b364-4c98-9d6a-873dc1dc3e7b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(86929df5-11f4-47e6-b556-4f68fb98918f),volume_id=86929df5-11f4-47e6-b556-4f68fb98918f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:48.615814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ac3a4468-90c2-4f73-8be8-a9d6885cd8dd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:06:48.623045 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:06:48.637369 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 46, 38926)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:48.637912 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:48.638347 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:48.638795 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (86929df5-11f4-47e6-b556-4f68fb98918f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:48.639251 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.79,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:46.038926 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:48.640361 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 324.6, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:06:48.642006 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:06:48.644913 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:06:48.648851 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:06:48.653178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:48.653178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:48.653178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.26, total_capacity_gb: 19.79,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:46.038926] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:48.653178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Weigher CapacityWeigher returned, weigher value is {max: 390.79999999999995, min: 390.79999999999995} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:48.653467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:48.653467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:06:48.653467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.79,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:48.643016 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:48.673042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ac3a4468-90c2-4f73-8be8-a9d6885cd8dd) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:06:48.675801 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-79b15534-226e-4fe3-9e34-f12fb77896f8 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Flow 'volume_create_scheduler' (570c1b9a-8e89-4862-b7d9-0c8cc7634cf9) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:06:49.912938 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.26, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 6, 46, 38926)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:06:49.914726 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:06:49.915242 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:49.915767 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:06:49.916479 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.79,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:48.643016 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:06:49.917811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:49.918315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:06:49.918886 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.26, total_capacity_gb: 19.79,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:48.643016] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:06:49.919934 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Weigher CapacityWeigher returned, weigher value is {max: 388.79999999999995, min: 388.79999999999995} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:06:49.920447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:06:49.921832 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-26c48a32-ffc5-4075-98f5-810cdba50b8f tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.260000000000002, total_capacity_gb: 19.79,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:06:49.920653 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:06:52.980540 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ae77391d-3928-4bb4-bdf8-605cf0a74f26 tempest-VolumesSnapshotNegativeTestJSON-1903441208 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:06:55.981080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cf80fb1b-996c-4f7b-a9ca-ec2cdebe6ba2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.79, 'free_capacity_gb': 19.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:07:14.574912 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-a6e45c26-9161-4edc-a810-1296b9747597 req-71deabd9-272f-45da-8da9-031ac392f140 tempest-ServersTestBootFromVolume-308351442 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:07:40.383107 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 7, 14, 570329)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:07:40.383905 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:07:40.384126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.384481 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:07:40.384864 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.385221 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.385597 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:14.570329] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:07:40.386183 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:07:40.386487 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:07:40.393288 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.24, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 7, 14, 570329)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:07:40.393704 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:07:40.394659 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.395022 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:07:40.395364 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.395717 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:40.396543 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.24, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:14.570329] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:07:40.396543 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:07:40.396814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:07:40.397145 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-e75f1d7b-15d3-48b0-b42e-c53bd69512a9 tempest-GroupsV320Test-1024203526 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:07:42.298681 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0656e262-2c1a-474f-9f9b-0994eb5c5221 tempest-GroupsV320Test-1024203526 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:07:45.851674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ae5fded0-0979-45dd-bb10-73ed8760bc75 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:07:49.766143 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Flow 'volume_create_scheduler' (e977b374-daa2-46e4-8161-772c4c248268) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:07:49.775457 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7dfd0b2e-a902-4d14-9fdd-fc40ec4fcc67) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:07:49.780609 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7dfd0b2e-a902-4d14-9fdd-fc40ec4fcc67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8ff8536d-b5c5-4945-b33d-fec35059e2d7),volume_id=8ff8536d-b5c5-4945-b33d-fec35059e2d7,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:07:49.787287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (351d67eb-50d6-4767-afed-591b9bdf9d07) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:07:49.800570 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:07:49.830763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 7, 42, 292874)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:07:49.831877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:07:49.832675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:49.833139 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (8ff8536d-b5c5-4945-b33d-fec35059e2d7) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:07:49.833642 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:42.292874 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 385.20000000000005, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:07:49.834883 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 345.20000000000005, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:07:49.836825 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:07:49.838234 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:07:49.839842 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:42.292874] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:07:49.841253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:07:49.841796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:07:49.841796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:07:49.841796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:49.839062 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:07:49.889122 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (351d67eb-50d6-4767-afed-591b9bdf9d07) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:07:49.891240 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f49576df-013f-4497-8793-c378585f11dd tempest-VolumesServicesTestJSON-651050963 None] Flow 'volume_create_scheduler' (e977b374-daa2-46e4-8161-772c4c248268) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:07:51.950765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c8ed2fe7-b70e-42b1-9c58-c732a6c794d8 tempest-VolumesServicesTestJSON-651050963 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:07:56.049259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b972101b-c119-441b-98dd-e96396acd482 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:07:59.591070 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Flow 'volume_create_scheduler' (68907b48-8c9e-420b-a096-b09f1e06a5fa) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:07:59.602439 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fdca3d51-09c2-42ed-af97-2c007c8b0019) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:07:59.623177 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (fdca3d51-09c2-42ed-af97-2c007c8b0019) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2f75fa2d-f67b-4d3a-baf0-a910d2b1b316),volume_id=2f75fa2d-f67b-4d3a-baf0-a910d2b1b316,volume_properties=VolumeProperties,volume_type=VolumeType(da0f45c8-69a9-4744-b796-e7a2f067456f))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:07:59.626185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4b38965b-a4e4-4582-818c-3f4c2ef4f1be) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:07:59.638179 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:07:59.655054 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 7, 56, 37448)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:07:59.655658 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:07:59.656433 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:59.656866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2f75fa2d-f67b-4d3a-baf0-a910d2b1b316) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:07:59.657391 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:56.037448 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:07:59.668183 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:59.668738 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:07:59.669086 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:56.037448] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:07:59.669518 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:07:59.669880 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:07:59.670431 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:07:59.670781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:07:59.670399 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:07:59.729287 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4b38965b-a4e4-4582-818c-3f4c2ef4f1be) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:07:59.732114 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6fc13eaa-d2e2-48a7-a23f-35a909993b3b tempest-VolumeTypesAccessTest-1523917369 None] Flow 'volume_create_scheduler' (68907b48-8c9e-420b-a096-b09f1e06a5fa) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:02.057493 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2ef8fba4-ea62-4005-89ef-3464aa5927e9 tempest-VolumeTypesAccessTest-1523917369 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:08:11.683695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Flow 'volume_create_scheduler' (a4b452ac-1116-4cf2-8a5e-3509bae74d96) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:11.686303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0cc14181-716b-4343-ae13-54cf2206a258) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:11.688856 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0cc14181-716b-4343-ae13-54cf2206a258) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(94ea4b5e-28a8-44a0-8c30-40ef94260806),volume_id=94ea4b5e-28a8-44a0-8c30-40ef94260806,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:11.691243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d34d1310-c400-4b88-a9f6-449e4ed42f97) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:11.700037 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:08:11.741956 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 2, 51421)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:11.742406 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:11.742952 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:11.743478 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (94ea4b5e-28a8-44a0-8c30-40ef94260806) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:11.746097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:02.051421 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:11.746700 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:08:11.748502 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:11.750494 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:11.752097 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:11.753513 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:11.753513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:11.753513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:11.753513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:02.051421] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:11.753513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:11.753862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:11.753862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:08:11.753862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:11.752435 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:11.812617 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d34d1310-c400-4b88-a9f6-449e4ed42f97) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:11.816048 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9fcf39bb-685e-45d9-8d42-0cfa389d9809 tempest-VolumesExtendTest-445567556 None] Flow 'volume_create_scheduler' (a4b452ac-1116-4cf2-8a5e-3509bae74d96) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:14.196398 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 2, 51421)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:14.199238 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:14.199621 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:14.199997 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Checking if host ncdv-master-1@ceph#ceph can extend the volume 94ea4b5e-28a8-44a0-8c30-40ef94260806 in 1 GB {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:51}} Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:11.752435 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:14.200391 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:08:14.209968 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:14.211640 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:14.214621 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 471, in extend_volume Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: backend_state = self.driver.backend_passes_filters( Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:14.218172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:14.219026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:11.752435] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:14.219026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:14.219026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:14.219026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a4dee0b4-3f6a-4434-b52f-f3b9f7add24a tempest-VolumesExtendTest-445567556 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:14.211198 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:15.829403 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Flow 'volume_create_scheduler' (fd29298f-1a00-44fa-9caa-deab01dec84e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:15.833490 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (244f3ce8-a438-4ce8-9ed9-825d8f48c511) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:15.836937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (244f3ce8-a438-4ce8-9ed9-825d8f48c511) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a9acc23a-f8c7-44f2-8c01-a4da542c1d75),volume_id=a9acc23a-f8c7-44f2-8c01-a4da542c1d75,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:15.839348 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e7ed7ffe-750a-4659-8cb3-b0cfe49d8ae1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:15.852969 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:08:15.870031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 2, 51421)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:15.870803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:15.871369 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:15.871877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a9acc23a-f8c7-44f2-8c01-a4da542c1d75) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:15.872561 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:14.211198 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:15.874387 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:15.874967 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:15.875498 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:14.211198] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:15.876361 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Weigher CapacityWeigher returned, weigher value is {max: 389.4, min: 389.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:15.876857 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:15.877501 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:08:15.878043 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:15.877686 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:15.933560 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e7ed7ffe-750a-4659-8cb3-b0cfe49d8ae1) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:15.941937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-f21c1a7c-bf67-469c-a059-d3a0d444715c tempest-VolumesExtendTest-445567556 None] Flow 'volume_create_scheduler' (fd29298f-1a00-44fa-9caa-deab01dec84e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:17.247486 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 2, 51421)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:17.248849 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:17.257958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:17.260285 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:17.260904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:15.877686 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:17.262281 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:08:17.264009 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:17.265395 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:17.268089 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:17.274937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:17.275754 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:17.279331 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:15.877686] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:17.287009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:17.287494 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:17.288013 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f32d9c04-883d-438c-8503-7c4e026795d0 tempest-VolumesExtendTest-445567556 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.329999999999998, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:17.287629 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:18.638711 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 2, 51421)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:18.646354 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:18.648756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:18.649293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Checking if host ncdv-master-1@ceph#ceph can extend the volume a9acc23a-f8c7-44f2-8c01-a4da542c1d75 in 1 GB {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:51}} Dec 11 21:08:18.650970 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.329999999999998, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:17.287629 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:18.655868 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:18.656494 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:18.657462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.329999999999998, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:17.287629] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:18.658798 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:18.661224 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:18.661826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8a033643-21be-478d-a3fc-55ed84377382 tempest-VolumesExtendTest-445567556 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.329999999999998, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:18.661448 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:22.238414 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-aa749a71-f9b1-4412-a07d-8882007e3906 tempest-VolumesExtendTest-445567556 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:08:24.523804 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cc06a4e4-84d9-49e7-b839-84af7ae9fa24 tempest-VolumesExtendTest-445567556 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:08:35.814721 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Flow 'volume_create_scheduler' (8fe5046c-e7c0-4471-8502-5b7282838057) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:35.832270 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (05e2dd35-5370-447a-90b2-7630376a6b2c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:35.835024 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (05e2dd35-5370-447a-90b2-7630376a6b2c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(af62f7be-16aa-4481-8b27-734592d4b6d5),volume_id=af62f7be-16aa-4481-8b27-734592d4b6d5,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:35.843265 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (61cbd7ab-f639-49e7-8f2c-9152f36b7139) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:35.865550 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:08:35.910778 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 24, 518441)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:35.911615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:35.912283 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:35.912858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (af62f7be-16aa-4481-8b27-734592d4b6d5) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:35.913610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:24.518441 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:35.914896 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 471, in extend_volume Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: backend_state = self.driver.backend_passes_filters( Dec 11 21:08:35.916392 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 306.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:35.918220 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:08:35.920589 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:08:35.922073 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:08:35.922073 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:08:35.926188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:35.926868 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:35.927446 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:24.518441] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:35.928608 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:35.929176 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:35.930097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:08:35.930765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:35.930324 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:35.983009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (61cbd7ab-f639-49e7-8f2c-9152f36b7139) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:35.999869 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-e863c800-4f88-462a-9313-8b860da7b835 req-69eb4629-33f4-4612-bee8-2ee40a10caf2 tempest-VolumesGetTestJSON-566803659 None] Flow 'volume_create_scheduler' (8fe5046c-e7c0-4471-8502-5b7282838057) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:38.007202 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-a42dc999-e9fb-401e-a963-78a349634bc4 req-be0e2986-fa72-4346-b9f6-d10916559ff8 tempest-VolumesGetTestJSON-566803659 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:08:42.230769 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Flow 'volume_create_scheduler' (50bdf0c8-f28f-44a6-a0f5-906f997327ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:42.236166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (23b076d4-10b4-4001-a825-80da5cc675ee) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:42.239923 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (23b076d4-10b4-4001-a825-80da5cc675ee) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2d7dc747-161e-49e9-961a-1a97814210ee),volume_id=2d7dc747-161e-49e9-961a-1a97814210ee,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:42.243752 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7fbe4ad9-51d8-4260-b3f2-12566ad85077) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:08:42.253111 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:08:42.300568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 38, 769)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:42.301163 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:42.301765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:42.302639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2d7dc747-161e-49e9-961a-1a97814210ee) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:42.303419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:38.000769 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:42.305488 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:08:42.307224 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:08:42.308929 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:08:42.310614 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:08:42.315065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:42.315711 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:42.316307 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:38.000769] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:42.316964 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:42.317525 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:42.318233 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:08:42.319069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:42.318462 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:42.378056 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7fbe4ad9-51d8-4260-b3f2-12566ad85077) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:08:42.382415 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-a35c794e-aad3-4f6f-a568-1e250a82e554 req-02ce3e48-7787-4acf-8029-f14494b07958 tempest-VolumesSnapshotsTestJSON-440060869 None] Flow 'volume_create_scheduler' (50bdf0c8-f28f-44a6-a0f5-906f997327ec) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:08:43.685774 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 8, 38, 769)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:08:43.686385 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:08:43.688108 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:43.689108 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:08:43.689594 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:42.318462 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:08:43.700979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:43.700979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:08:43.700979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:42.318462] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:08:43.700979 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:08:43.701633 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:08:43.701633 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-90794fe4-072f-4947-aeec-b61768020731 req-3b416d38-6d6d-4122-9323-cad1669d3bdd tempest-VolumesSnapshotsTestJSON-440060869 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:08:43.693041 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:08:45.854908 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-44e60b43-0bd8-415b-b75b-79fde80dde62 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:08:46.982280 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-42a5005b-d2c5-492a-96cc-dbaa01ece86e req-81597c0e-cc3f-48f6-b1f7-9feaa619cb1f tempest-VolumesSnapshotsTestJSON-440060869 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:08:55.994126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bff52841-7581-4ce9-9709-b695b1be2275 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:09:45.848070 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7ea409f6-6a5e-4545-aafe-c5d11e746d2d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:09:55.935477 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-06617c45-b896-4335-9f82-dba47bc33513 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:08.503470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-650f3d63-8446-4f23-8486-d14ccf7884c3 tempest-BackendsCapabilitiesAdminTestsJSON-2036315772 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 9, 55, 930815)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:08.541322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-57701404-5371-4b00-ae26-5ce0e80ee7cb tempest-BackendsCapabilitiesAdminTestsJSON-2036315772 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 9, 55, 930815)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:12.716840 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Flow 'volume_create_scheduler' (ca80295f-e66f-46b6-8514-ccb06406b7a5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:12.720547 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (396cec4f-6959-4af6-aee8-f58fe0c7f5bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:12.723261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (396cec4f-6959-4af6-aee8-f58fe0c7f5bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a44c9c41-d379-4d4f-aed8-1ca2ced23613),volume_id=a44c9c41-d379-4d4f-aed8-1ca2ced23613,volume_properties=VolumeProperties,volume_type=VolumeType(b72cc19d-bd3f-48e8-909b-b79d3332d424))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:12.725274 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (473ddcf3-a9d1-42fd-93e1-47cad79bdcce) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:12.732288 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:12.749551 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 9, 55, 930815)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:12.749970 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:12.750653 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:12.751065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a44c9c41-d379-4d4f-aed8-1ca2ced23613) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:12.751468 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:12.752687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:12.754909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:12.756906 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:12.759225 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Volume type extra spec requirement "storage_protocol=tempest-storage_protocol-1335017360" does not match reported capability "ceph" {{(pid=133283) _satisfies_extra_specs /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:86}} Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815 fails resource_type extra_specs requirements {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:100}} Dec 11 21:10:12.761184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Filter CapabilitiesFilter returned 0 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:12.761653 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Filtering removed all hosts for the request with volume ID 'a44c9c41-d379-4d4f-aed8-1ca2ced23613'. Filter results: [('AvailabilityZoneFilter', ['ncdv-master-1@ceph#ceph']), ('CapacityFilter', ['ncdv-master-1@ceph#ceph']), ('CapabilitiesFilter', [])] {{(pid=133283) _log_filtration /opt/stack/cinder/cinder/scheduler/base_filter.py:73}} Dec 11 21:10:12.761653 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.scheduler.base_filter [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Filtering removed all hosts for the request with volume ID 'a44c9c41-d379-4d4f-aed8-1ca2ced23613'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0) Dec 11 21:10:12.761653 ncdv-master-1 cinder-scheduler[133283]: WARNING cinder.scheduler.filter_scheduler [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] No weighed backend found for volume with properties: {'id': 'b72cc19d-bd3f-48e8-909b-b79d3332d424', 'name': 'tempest-UserMessagesTest-volume-type-1470978734', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'tempest-storage_protocol-1335017360', 'vendor_name': 'tempest-vendor_name-116494870'}, 'qos_specs_id': None, 'created_at': '2021-12-11T21:10:12.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False} Dec 11 21:10:12.761653 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.message.api [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Creating message record for request_id = req-6e783240-30b6-4a34-b656-9f7eb60105e8 Dec 11 21:10:12.772078 ncdv-master-1 cinder-scheduler[133283]: ERROR cinder.scheduler.flows.create_volume [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Dec 11 21:10:12.772078 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.volume.flows.common [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Setting Volume a44c9c41-d379-4d4f-aed8-1ca2ced23613 to error due to: No valid backend was found. No weighed backends available {{(pid=133283) error_out /opt/stack/cinder/cinder/volume/flows/common.py:81}} Dec 11 21:10:12.787227 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (473ddcf3-a9d1-42fd-93e1-47cad79bdcce) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:12.791107 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-6e783240-30b6-4a34-b656-9f7eb60105e8 tempest-UserMessagesTest-614522858 None] Flow 'volume_create_scheduler' (ca80295f-e66f-46b6-8514-ccb06406b7a5) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:14.148916 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Flow 'volume_create_scheduler' (0a3c672b-c374-4e79-850a-551410bf535f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:14.151786 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d480d11-bbd5-4b70-9f6f-d54d9682ce53) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:14.155756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1d480d11-bbd5-4b70-9f6f-d54d9682ce53) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1475c40d-ac25-4ca2-aae4-e6a651bb9922),volume_id=1475c40d-ac25-4ca2-aae4-e6a651bb9922,volume_properties=VolumeProperties,volume_type=VolumeType(9560b69a-282c-456f-aa2f-f9db9fc4b59a))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:14.159585 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (32266586-dd8c-4fe9-8bac-63f66305484a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:14.168717 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:14.187562 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 9, 55, 930815)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:14.187974 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:14.188455 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:14.189938 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1475c40d-ac25-4ca2-aae4-e6a651bb9922) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:14.190518 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:14.191952 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:14.193629 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:14.203853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:14.205845 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:14.207803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:14.207803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Volume type extra spec requirement "storage_protocol=tempest-storage_protocol-1476176479" does not match reported capability "ceph" {{(pid=133283) _satisfies_extra_specs /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:86}} Dec 11 21:10:14.207803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capabilities_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815 fails resource_type extra_specs requirements {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capabilities_filter.py:100}} Dec 11 21:10:14.207803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Filter CapabilitiesFilter returned 0 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:14.208135 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Filtering removed all hosts for the request with volume ID '1475c40d-ac25-4ca2-aae4-e6a651bb9922'. Filter results: [('AvailabilityZoneFilter', ['ncdv-master-1@ceph#ceph']), ('CapacityFilter', ['ncdv-master-1@ceph#ceph']), ('CapabilitiesFilter', [])] {{(pid=133283) _log_filtration /opt/stack/cinder/cinder/scheduler/base_filter.py:73}} Dec 11 21:10:14.208135 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.scheduler.base_filter [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Filtering removed all hosts for the request with volume ID '1475c40d-ac25-4ca2-aae4-e6a651bb9922'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0) Dec 11 21:10:14.208135 ncdv-master-1 cinder-scheduler[133283]: WARNING cinder.scheduler.filter_scheduler [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] No weighed backend found for volume with properties: {'id': '9560b69a-282c-456f-aa2f-f9db9fc4b59a', 'name': 'tempest-UserMessagesTest-volume-type-1319103874', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'storage_protocol': 'tempest-storage_protocol-1476176479', 'vendor_name': 'tempest-vendor_name-755778943'}, 'qos_specs_id': None, 'created_at': '2021-12-11T21:10:14.000000', 'updated_at': None, 'deleted_at': None, 'deleted': False} Dec 11 21:10:14.208135 ncdv-master-1 cinder-scheduler[133283]: INFO cinder.message.api [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Creating message record for request_id = req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 Dec 11 21:10:14.218390 ncdv-master-1 cinder-scheduler[133283]: ERROR cinder.scheduler.flows.create_volume [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Dec 11 21:10:14.218984 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.volume.flows.common [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Setting Volume 1475c40d-ac25-4ca2-aae4-e6a651bb9922 to error due to: No valid backend was found. No weighed backends available {{(pid=133283) error_out /opt/stack/cinder/cinder/volume/flows/common.py:81}} Dec 11 21:10:14.229804 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (32266586-dd8c-4fe9-8bac-63f66305484a) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:14.232779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-bcd5fc77-a458-4a99-9e15-93c04f52ec32 tempest-UserMessagesTest-614522858 None] Flow 'volume_create_scheduler' (0a3c672b-c374-4e79-850a-551410bf535f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:24.723305 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Flow 'volume_create_scheduler' (909d069e-63e5-4ba6-8d7e-3207fbb73896) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:24.725719 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (849a6809-8547-4403-bbc7-68d9a48c167e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:24.728169 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (849a6809-8547-4403-bbc7-68d9a48c167e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a04dad6e-9d40-4352-ac96-075fe6c9bca8),volume_id=a04dad6e-9d40-4352-ac96-075fe6c9bca8,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:24.730130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7ceadb31-22be-4d9c-9eb6-989ec4eb2fc4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:24.737392 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:24.755278 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 9, 55, 930815)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:24.755697 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:24.756060 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:24.756419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a04dad6e-9d40-4352-ac96-075fe6c9bca8) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:24.756815 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:24.758210 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:24.758622 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:24.758995 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:09:55.930815] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:24.759589 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:24.759937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:24.760407 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:24.760781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:24.760431 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:24.791615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7ceadb31-22be-4d9c-9eb6-989ec4eb2fc4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:24.793797 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-0199f4ea-7802-410a-a6c7-e5cd5248f76d tempest-VolumesMetadataTest-2014252962 None] Flow 'volume_create_scheduler' (909d069e-63e5-4ba6-8d7e-3207fbb73896) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:26.886626 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4d16cb2b-7829-45b2-afbc-07a3ced2ebf5 tempest-VolumesMetadataTest-2014252962 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:30.346438 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Flow 'volume_create_scheduler' (a0ad807e-828f-4de6-8f55-d12bc11889c6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:30.349214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3a29b320-8b12-4cf6-a19b-e4ab812190b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:30.351652 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (3a29b320-8b12-4cf6-a19b-e4ab812190b1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d11c3d7a-08a9-4b14-836b-268cb8d2af62),volume_id=d11c3d7a-08a9-4b14-836b-268cb8d2af62,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:30.353768 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ca8476c7-7b1e-4c3e-819c-20fd5a8cb106) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:30.361391 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:30.378779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 26, 882278)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:30.379321 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:30.379895 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:30.380670 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d11c3d7a-08a9-4b14-836b-268cb8d2af62) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:30.381247 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:26.882278 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:30.382450 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:30.384473 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:30.386014 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:30.390132 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:26.882278] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:30.393065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:30.393534 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:30.393534 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:30.393534 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:30.389665 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:30.416071 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ca8476c7-7b1e-4c3e-819c-20fd5a8cb106) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:30.418045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-ebdc7250-75c3-4221-8e94-8f55f79095da tempest-VolumesTransfersV355Test-1412325425 None] Flow 'volume_create_scheduler' (a0ad807e-828f-4de6-8f55-d12bc11889c6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:33.064988 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-76bd9dc6-8b83-4526-b1b8-ac19e3a47e50 tempest-VolumesTransfersV355Test-295471338 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:33.900440 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Flow 'volume_create_scheduler' (c2a02577-c1a3-44a6-afdb-d9635801a8b1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:33.903016 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e7796f91-2f56-4f09-8fe8-a94cb8a4d8ab) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:33.912366 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e7796f91-2f56-4f09-8fe8-a94cb8a4d8ab) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(812b0af1-dc10-487e-bee7-144d9c2c1ab5),volume_id=812b0af1-dc10-487e-bee7-144d9c2c1ab5,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:33.914315 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2e80d1ea-ab0c-4bf1-8f5a-43b8d6274d13) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:33.922057 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:33.937438 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 33, 60591)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:33.937953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:33.938322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:33.938709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (812b0af1-dc10-487e-bee7-144d9c2c1ab5) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:33.939102 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:33.060591 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:33.940145 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:33.942029 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:33.943782 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:33.945864 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:33.947739 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:33.947739 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:33.947739 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:33.947739 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:33.060591] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:33.947739 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:33.948166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:33.948166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:33.948166 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:33.942460 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:33.972447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (2e80d1ea-ab0c-4bf1-8f5a-43b8d6274d13) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:33.974550 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9f755d77-7528-42f7-82e5-b2b3855b9154 tempest-VolumesTransfersV355Test-1412325425 None] Flow 'volume_create_scheduler' (c2a02577-c1a3-44a6-afdb-d9635801a8b1) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:35.893777 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2147dd78-237f-47e5-82c9-0249dc978857 tempest-VolumesTransfersV355Test-295471338 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:39.942932 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (ca2fcd09-8e7a-4d6f-bf95-6880fb44b455) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:39.945388 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ee8d8d87-eb34-4adc-b409-e5aca908517f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:39.947888 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (ee8d8d87-eb34-4adc-b409-e5aca908517f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(aad251d9-75e3-45be-b9ca-acfd013de8de),volume_id=aad251d9-75e3-45be-b9ca-acfd013de8de,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:39.950391 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6bab1bd4-c939-4e7f-873b-a5047aebf974) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:39.958603 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:39.973066 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 35, 888469)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:39.973488 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:39.973863 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:39.974226 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (aad251d9-75e3-45be-b9ca-acfd013de8de) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:39.974650 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:35.888469 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:39.975732 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:39.977645 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:39.981345 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:39.984209 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:39.986089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:39.986089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:39.986089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:35.888469] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:39.986089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:39.986089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:39.987879 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:39.987879 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:39.978139 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:40.005756 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6bab1bd4-c939-4e7f-873b-a5047aebf974) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:40.007788 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-97e6fac6-4f23-4cc8-8c5c-d5e3869ca7a5 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (ca2fcd09-8e7a-4d6f-bf95-6880fb44b455) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:41.454929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (e752b9a3-3d70-4257-aa59-f227e4ac0aca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:41.457402 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (47d7c52f-726c-4c5f-99af-7c8dacce4913) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:41.459808 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (47d7c52f-726c-4c5f-99af-7c8dacce4913) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2b786c6d-f629-4a63-874c-fc196308a591),volume_id=2b786c6d-f629-4a63-874c-fc196308a591,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:41.461707 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0b69e1ee-b489-4d7b-a81d-66603be755bc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:41.469455 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:41.489921 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 35, 888469)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:41.490445 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:41.490851 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:41.491213 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2b786c6d-f629-4a63-874c-fc196308a591) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:41.491616 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:39.978139 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:41.493035 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:41.493683 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:41.494086 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:39.978139] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:41.494524 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:41.494897 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:41.495372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:41.495750 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:41.495399 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:41.521926 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0b69e1ee-b489-4d7b-a81d-66603be755bc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:41.524682 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3d7bfb08-6825-4201-947f-7dd241cbed07 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (e752b9a3-3d70-4257-aa59-f227e4ac0aca) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:43.103646 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1c075841-10d1-409a-857c-7fbfb1ba38b6 tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:44.262948 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21d229fd-1a34-464d-8a7b-cd8e895b25c8 tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:45.097212 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (173ef209-fb0e-4471-92e1-51039951c53d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:45.100075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6f7c3023-52a4-4e54-b085-bc1f4bac286f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:45.102718 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6f7c3023-52a4-4e54-b085-bc1f4bac286f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fc699323-3917-4d3c-b132-a2862f4abb24),volume_id=fc699323-3917-4d3c-b132-a2862f4abb24,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:45.104632 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (403f3ee8-864a-4236-8556-21e41f3a6ae8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:45.111617 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:45.126722 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 44, 258512)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:45.127139 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:45.127511 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:45.127879 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (fc699323-3917-4d3c-b132-a2862f4abb24) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:45.128278 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:44.258512 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:45.129321 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:45.131279 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:45.132802 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:45.134408 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:44.258512] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:45.139045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:45.139576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:45.139576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:45.139576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:45.131856 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:45.170197 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (403f3ee8-864a-4236-8556-21e41f3a6ae8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:45.172228 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-99fd7408-b506-4786-b758-a89b6fb56d6b tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (173ef209-fb0e-4471-92e1-51039951c53d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:45.845589 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fa18f372-7a19-4415-bd0f-2da362226f93 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:10:46.497825 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (3cbab10a-6322-4a08-ad77-90d7bc8122f0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:46.503049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (49613e39-2959-45ad-a551-017c5e91feeb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:46.506516 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (49613e39-2959-45ad-a551-017c5e91feeb) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=None,source_replicaid=,source_volid=fc699323-3917-4d3c-b132-a2862f4abb24,volume=Volume(b63ea72f-6e49-413d-9030-ec1d8334f541),volume_id=b63ea72f-6e49-413d-9030-ec1d8334f541,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:46.509113 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8d933fad-9970-4524-a3d7-556205f78284) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:46.523421 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:46.540048 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 44, 258512)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:46.540590 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:46.541054 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:46.541514 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b63ea72f-6e49-413d-9030-ec1d8334f541) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:46.542009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:45.131856 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:46.543418 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:46.545029 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:46.546847 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:46.549058 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:45.131856] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:46.551657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:46.552146 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:46.552146 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:46.549629 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:46.578835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (8d933fad-9970-4524-a3d7-556205f78284) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:46.580929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c7ffaa67-3818-4acd-ace4-e6b6368bc5e4 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (3cbab10a-6322-4a08-ad77-90d7bc8122f0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:49.034906 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (e3f659b3-2073-48f0-b70b-c134b707a26a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:49.038070 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (40692420-ba51-43ab-a36e-22fb044276bf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:49.040691 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (40692420-ba51-43ab-a36e-22fb044276bf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(36906ec8-dd71-440e-b668-e734e7a42b6d),volume_id=36906ec8-dd71-440e-b668-e734e7a42b6d,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:49.044591 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (13fd3665-71a2-4d6c-8ed4-f138a4f139c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:49.054621 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:49.069536 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 44, 258512)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:49.069982 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:49.070354 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:49.070745 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (36906ec8-dd71-440e-b668-e734e7a42b6d) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:49.071149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:46.549629 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:49.072248 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:49.073982 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:49.076621 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:49.078369 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:49.080126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:49.080126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:49.080126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:46.549629] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:49.080126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 389.4, min: 389.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:49.080126 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:49.080478 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:49.080478 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:49.075192 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:49.101718 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (13fd3665-71a2-4d6c-8ed4-f138a4f139c4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:49.103784 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1705a268-9a3a-4917-8df5-2b7c9d58f2c1 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (e3f659b3-2073-48f0-b70b-c134b707a26a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:50.748294 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6778cb0f-a859-4157-b8af-abe303d5dc30 tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:52.790050 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-349e0a5a-285b-4f29-8847-14f9c32efa20 tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:53.922635 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (d10c0dc2-4098-4d94-98d4-81d03b1f0324) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:53.925081 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a63bb9bc-4326-4d7d-ab7a-d12b88ef152b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:53.927484 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a63bb9bc-4326-4d7d-ab7a-d12b88ef152b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(25f6a191-14f9-4501-a950-6f905b5b1ea5),volume_id=25f6a191-14f9-4501-a950-6f905b5b1ea5,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:53.929389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (22a2039b-79e1-4368-9df1-c7649b17b5b5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:53.937537 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:53.959318 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 52, 785947)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:53.959741 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:53.960111 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:53.960475 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (25f6a191-14f9-4501-a950-6f905b5b1ea5) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:53.960871 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:52.785947 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:53.962266 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:53.962685 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:53.963063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:52.785947] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:53.963481 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:53.963831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:53.964306 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:53.965210 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:53.964334 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:53.991988 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (22a2039b-79e1-4368-9df1-c7649b17b5b5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:53.994069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-56914b74-f0eb-4771-9a38-4c103d963e51 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (d10c0dc2-4098-4d94-98d4-81d03b1f0324) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:55.420358 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (d649350d-99ae-4107-aa1a-22d121e23795) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:55.422796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5eb40a7e-112a-4764-9c07-5ab4abafca44) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:55.425191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (5eb40a7e-112a-4764-9c07-5ab4abafca44) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(57e75a06-814f-4ce7-8721-1f44446cbdb5),volume_id=57e75a06-814f-4ce7-8721-1f44446cbdb5,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:55.427133 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f14b9a59-d0d7-438b-b61a-3ff2ba6af9a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:10:55.435193 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:10:55.455545 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 52, 785947)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:10:55.456154 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:10:55.456702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:55.457069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (57e75a06-814f-4ce7-8721-1f44446cbdb5) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:10:55.457476 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:53.964334 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:55.458537 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:10:55.460960 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:10:55.462944 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:10:55.465437 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:53.964334] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:10:55.467715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Weigher CapacityWeigher returned, weigher value is {max: 389.4, min: 389.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:10:55.468223 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:10:55.468223 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:10:55.468223 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:55.462042 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:10:55.490977 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (f14b9a59-d0d7-438b-b61a-3ff2ba6af9a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:10:55.492974 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-901d97bf-9014-48c0-a2fc-ae8874bab7d7 tempest-VolumesGetTest-1016277363 None] Flow 'volume_create_scheduler' (d649350d-99ae-4107-aa1a-22d121e23795) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:10:55.918584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bc4ca1f2-2f2d-4077-abf5-762233ac9dd1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:57.123009 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f5f50f3c-3ae4-47ab-ba93-485f98d8f55f tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:58.802591 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9385f9b3-1ab6-4c71-9c84-5901a12ead08 tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:10:59.333006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-13c43827-dc9f-4a7b-babf-f635cd865deb tempest-VolumesGetTest-1016277363 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:11:02.886882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (06a2ff1b-d725-450b-9866-2503ac953439) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:02.890462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (92b5403a-d137-4b29-ac1c-5ec3fb044a35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:02.892920 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (92b5403a-d137-4b29-ac1c-5ec3fb044a35) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a5ad3712-ba49-4336-9ea6-4de8b933fc6f),volume_id=a5ad3712-ba49-4336-9ea6-4de8b933fc6f,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:02.896215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (12701665-8d84-4be4-8632-752b833378f5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:02.904384 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:11:02.918762 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:02.919204 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:02.919585 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:02.919963 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a5ad3712-ba49-4336-9ea6-4de8b933fc6f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:02.920373 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:59.328792 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:11:02.921430 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:02.923227 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:11:02.925074 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:11:02.927344 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:11:02.929027 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:11:02.929027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:02.929027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:02.929027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:10:59.328792] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:02.929027 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 391.4, min: 391.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:02.929435 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:02.929435 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:11:02.929435 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:02.923862 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:02.951779 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (12701665-8d84-4be4-8632-752b833378f5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:02.953836 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-7e8dc404-b873-4fed-9361-d086ca273091 tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (06a2ff1b-d725-450b-9866-2503ac953439) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:04.112558 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:04.113211 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:04.113336 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:04.113671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:04.114117 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:02.923862 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:04.115177 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:04.116007 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:04.116462 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:02.923862] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:04.116891 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 390.4, min: 390.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:04.117299 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:04.117703 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f16b59d-84e9-4653-957a-8fb30a6873dd tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:04.117349 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:05.326708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-f5ecd2a5-c97b-4af3-bc3c-8999ff9c4d2c tempest-VolumesSnapshotTestJSON-970988873 None] create_backup in rpcapi backup_id 1fa0cd39-35e6-4e2e-8601-5220ba11e46f {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:11:12.729278 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Flow 'volume_create_scheduler' (afe012b0-2036-4e29-b082-c9444ae5940b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:12.732402 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (962a8768-15ce-4f96-a5a7-41330abc2af6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:12.735589 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (962a8768-15ce-4f96-a5a7-41330abc2af6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(531e4493-0fa5-4b02-b9c7-c8db7e07cc42),volume_id=531e4493-0fa5-4b02-b9c7-c8db7e07cc42,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:12.737987 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2091cfc-1cf7-44d5-bb6d-d3cbfc86006b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:12.748701 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:11:12.784471 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:12.785051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:12.785590 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:12.786107 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (531e4493-0fa5-4b02-b9c7-c8db7e07cc42) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:12.786752 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:04.117349 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:12.788015 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:12.790111 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:12.792128 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:11:12.793712 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:11:12.795710 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:11:12.795710 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:11:12.797551 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:12.798762 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:12.799215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:04.117349] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:12.800195 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Weigher CapacityWeigher returned, weigher value is {max: 389.4, min: 389.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:12.801245 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:12.803351 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:11:12.803533 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:12.803161 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:12.866516 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d2091cfc-1cf7-44d5-bb6d-d3cbfc86006b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:12.868512 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-a66178f2-6c00-44a8-bd5f-a168e17c9f59 tempest-AttachVolumeShelveTestJSON-486918854 None] Flow 'volume_create_scheduler' (afe012b0-2036-4e29-b082-c9444ae5940b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:30.681067 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (c42736e1-8a9e-4232-8dcc-465065acc6b7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:30.698226 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c27b0a96-16f8-4bc7-9aa3-1cbb72318a63) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:30.703858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c27b0a96-16f8-4bc7-9aa3-1cbb72318a63) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d439c366-f8ce-44a8-806c-887821415191),volume_id=d439c366-f8ce-44a8-806c-887821415191,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:30.712303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9052265d-4e33-4ffa-966d-3e16ca85d684) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:11:30.729733 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:11:30.757843 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:30.758239 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:30.767420 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:30.768168 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d439c366-f8ce-44a8-806c-887821415191) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:30.768777 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:12.803161 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:11:30.770043 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:30.772044 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:11:30.775068 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:11:30.778283 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:11:30.784077 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:30.784807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:30.785364 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:12.803161] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:30.786036 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:30.786915 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:30.791334 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:11:30.792449 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:30.792065 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:30.859049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9052265d-4e33-4ffa-966d-3e16ca85d684) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:11:30.861045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18b96346-5e23-44f6-9a2f-23e0ff2f94bb tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (c42736e1-8a9e-4232-8dcc-465065acc6b7) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:11:36.789069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:36.794671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:36.797660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:36.798214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:36.798647 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:30.792065 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:36.799869 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:36.800234 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:36.800617 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:30.792065] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:36.801029 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:36.801833 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:36.801833 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d4d1573b-e573-4d8d-be74-31bc23c68f18 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:36.801433 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:39.145529 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:39.146096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:39.146872 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:39.148265 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:39.148265 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:36.801433 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:39.149232 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:39.149769 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:39.150264 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 14.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:36.801433] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:39.152314 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:39.153039 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:39.153674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-041dde55-0334-4c91-95c1-95b8cbf21143 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:39.153317 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:40.536891 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 10, 59, 328792)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:11:40.543312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:11:40.543700 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:40.544069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:11:40.544732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:39.153317 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:40.545521 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:40.547532 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 306.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:11:40.549315 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 286.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:11:40.553104 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:11:40.558207 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:40.558207 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:11:40.558207 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 13.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:39.153317] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:11:40.558207 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:11:40.558207 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:11:40.559128 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b240f775-6153-442f-9ac7-9e941a2025aa tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 12.32, total_capacity_gb: 19.77,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:11:40.547941 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:11:45.847141 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ffe4b6a1-8c5d-4808-a7e5-9787e1c6672c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:11:56.031389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c5c44da6-2105-4690-8d53-45b1097e4de0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:12:45.844490 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3ab311bc-5659-4455-8850-fcfc06ad7498 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:12:55.909884 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0fd77e8f-4da6-4b40-a278-1220743502c4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:13:22.064615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 12, 55, 905599)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:13:22.065327 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:13:22.065554 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:22.065937 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:13:22.066330 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:12:55.905599 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:13:22.067606 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:22.068028 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:22.068547 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:12:55.905599] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:13:22.069237 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:13:22.069559 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:13:22.069975 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1ff76983-befb-44b1-b618-78ef30fdd05c tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:22.069604 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:13:24.790266 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 12, 55, 905599)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:13:24.791237 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:13:24.791237 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:24.791636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:13:24.791636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:22.069604 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:13:24.792454 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 266.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:13:24.795201 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:13:24.799599 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:13:24.803756 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:13:24.810295 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:13:24.810295 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:24.810295 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:24.810295 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:22.069604] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:13:24.810295 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:13:24.811032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:13:24.811032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6d8b4618-eee6-4e52-9203-36761f9dd74d tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:24.794424 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:13:45.851496 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-41e90561-9318-4483-b754-ddde5446f16a None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:13:49.709370 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 12, 55, 905599)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:13:49.710882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:13:49.711419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:49.712046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:13:49.712591 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:24.794424 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:13:49.715343 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:49.715888 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:49.716588 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:24.794424] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:13:49.717420 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:13:49.718090 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:13:49.718675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ec8fdeaa-3dd7-4f47-9ae7-54167fadbde8 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:49.718295 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:13:52.158131 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 12, 55, 905599)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:13:52.158740 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:13:52.158946 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:52.159311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:13:52.159698 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:49.718295 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:13:52.162977 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:52.165918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:13:52.166499 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:49.718295] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:13:52.167101 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:13:52.167467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:13:52.167878 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-12ffe3df-22ca-430b-b49e-95ae1bcb7f83 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.77,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:52.167518 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:13:55.985064 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c06a0f1e-2e89-4972-9ed9-8776d63e0240 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:14:17.078410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Flow 'volume_create_scheduler' (872271fa-f53c-4be9-9571-599d061deeed) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:17.080983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (15ea46dc-4a22-40d3-b335-97cfe08b2409) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:17.084141 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (15ea46dc-4a22-40d3-b335-97cfe08b2409) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(973376c4-c2de-491a-8c04-190c2ffd5bac),volume_id=973376c4-c2de-491a-8c04-190c2ffd5bac,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:17.086241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f3cde31-9add-46d0-8dff-28e238aee198) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:17.094540 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:14:17.110502 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 13, 55, 980123)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:14:17.111109 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:14:17.111547 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:17.111974 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (973376c4-c2de-491a-8c04-190c2ffd5bac) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:14:17.112445 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:55.980123 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:14:17.113098 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:14:17.116382 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:14:17.127742 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:14:17.131254 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:13:55.980123] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:14:17.133428 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:14:17.134031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:14:17.134031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:14:17.134031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:17.119030 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:14:17.150267 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f3cde31-9add-46d0-8dff-28e238aee198) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:17.152284 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c0e63c81-9284-46da-8c29-f2fe449fde88 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Flow 'volume_create_scheduler' (872271fa-f53c-4be9-9571-599d061deeed) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:18.665075 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bf76ded5-a363-4018-8753-b8612a75e4d2 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:14:19.543689 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Flow 'volume_create_scheduler' (6173c6d8-d710-4ca0-8bfe-32a8ef240ce3) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:19.546066 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7baa9635-2dfc-4e2d-ac80-ed752f8fe32e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:19.548499 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7baa9635-2dfc-4e2d-ac80-ed752f8fe32e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9dd1b324-e373-4860-9f2b-f2ec2076cb63),volume_id=9dd1b324-e373-4860-9f2b-f2ec2076cb63,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:19.550389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e4fd7086-3af8-47c2-8815-413464dd69ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:19.557709 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:14:19.572326 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 18, 660742)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:14:19.572744 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:14:19.573112 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:19.573469 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9dd1b324-e373-4860-9f2b-f2ec2076cb63) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:14:19.573870 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:18.660742 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:14:19.575012 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:14:19.577531 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:14:19.579671 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:14:19.582852 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:14:19.585153 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:14:19.585153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:19.585153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:19.585153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:18.660742] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:14:19.585153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:14:19.585831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:14:19.585831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:14:19.585831 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:19.577381 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:14:19.605492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e4fd7086-3af8-47c2-8815-413464dd69ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:19.607807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-073846af-8262-4779-b7d5-4faaa4d8b794 tempest-VolumeQuotasAdminTestJSON-1311686007 None] Flow 'volume_create_scheduler' (6173c6d8-d710-4ca0-8bfe-32a8ef240ce3) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:21.825627 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d7afc039-0539-443d-b9bb-2195ada4a143 tempest-VolumeQuotasAdminTestJSON-1443030787 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:14:26.701783 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Flow 'volume_create_scheduler' (966e1ed9-fd45-4c6a-9d6b-4634d75fd60b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:26.704794 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b18f94a7-9768-4af6-a8ba-ce4102c29c8b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:26.707467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (b18f94a7-9768-4af6-a8ba-ce4102c29c8b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(ce273410-93ab-4b56-985b-4555243aba13),volume_id=ce273410-93ab-4b56-985b-4555243aba13,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:26.709857 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4d114eaf-9a2a-4ff7-892f-fb99cc1fc5bb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:26.716907 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:14:26.734074 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 21, 820257)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:14:26.734360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:14:26.734848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:26.735367 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (ce273410-93ab-4b56-985b-4555243aba13) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:14:26.735767 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:21.820257 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:14:26.736837 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:14:26.739645 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:14:26.742002 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:14:26.744447 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:14:26.747311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:26.747311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:26.747311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:21.820257] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:14:26.747311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:14:26.747311 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:14:26.747826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:14:26.747826 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:26.740401 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:14:26.771256 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4d114eaf-9a2a-4ff7-892f-fb99cc1fc5bb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:26.773377 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21c9a21a-7921-414e-a0d0-0d25c2269b94 tempest-VolumesTransfersTest-133636199 None] Flow 'volume_create_scheduler' (966e1ed9-fd45-4c6a-9d6b-4634d75fd60b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:28.970171 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0f374f24-ebc5-4028-90bf-9864e8858151 tempest-VolumesTransfersTest-1508651529 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:14:29.858176 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Flow 'volume_create_scheduler' (195b9523-be2f-46ca-8b49-f469ada4f584) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:29.860564 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9491476a-fb03-4fb3-aff1-941225e8344c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:29.862972 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (9491476a-fb03-4fb3-aff1-941225e8344c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(9d7005ce-ff21-4670-ad97-2173ba125e7e),volume_id=9d7005ce-ff21-4670-ad97-2173ba125e7e,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:29.865153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (dfa54351-4ecf-4a79-9f8d-16ffab186f27) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:14:29.874539 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:14:29.889829 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 28, 965962)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:14:29.889829 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:14:29.890293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:29.890821 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (9d7005ce-ff21-4670-ad97-2173ba125e7e) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:14:29.891421 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:28.965962 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:14:29.893338 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:29.894343 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:14:29.894950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:28.965962] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:14:29.895317 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:14:29.895667 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:14:29.896140 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:14:29.896515 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:29.896166 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:14:29.922380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (dfa54351-4ecf-4a79-9f8d-16ffab186f27) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:14:29.924323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-594ffee0-f33b-4966-b6d5-af19752399e4 tempest-VolumesTransfersTest-133636199 None] Flow 'volume_create_scheduler' (195b9523-be2f-46ca-8b49-f469ada4f584) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:14:31.635599 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7115bf40-5a9f-424a-b4d8-4681e489311c tempest-VolumesTransfersTest-1508651529 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:14:45.855709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-37f3b2ab-0014-4df5-8b4d-efb5777732f9 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:14:55.954492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ec694c54-2298-45fd-9059-461590daa93c None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:31.389944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (be4fb4f6-ad3a-42e8-8d09-cf3f2542aeda) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:31.394854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (496cff82-c849-43da-9699-f975d4bb72c0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:31.398028 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (496cff82-c849-43da-9699-f975d4bb72c0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(b242c9ed-ca39-4845-aad5-a78c6df1ed8c),volume_id=b242c9ed-ca39-4845-aad5-a78c6df1ed8c,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:31.401220 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (03a49320-d5b2-4b30-b294-f32e534a24c8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:31.430191 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:15:31.446898 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 55, 949565)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:31.447325 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:31.447694 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:31.448049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (b242c9ed-ca39-4845-aad5-a78c6df1ed8c) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:31.449024 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:55.949565 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:31.450252 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:15:31.452915 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:15:31.456613 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:31.458831 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:14:55.949565] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:31.463400 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:31.463958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:31.464250 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:15:31.464553 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:31.464206 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:31.518983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (03a49320-d5b2-4b30-b294-f32e534a24c8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:31.521014 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1684a91c-56cf-4998-ba03-b388bd54743e tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (be4fb4f6-ad3a-42e8-8d09-cf3f2542aeda) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:32.646784 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 55, 949565)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:32.647865 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:32.648235 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:32.648595 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:32.648973 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:31.464206 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:15:32.649794 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:15:32.651886 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:32.653687 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:32.657442 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:32.662859 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:32.662859 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:32.662859 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:31.464206] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:32.662859 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:32.662859 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:32.665409 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f244746-46c6-4f33-999f-92f1f4d5cf0b tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:32.651871 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:34.982090 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (79f1a7bc-7264-4248-812b-a5139f0b00ec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:34.984702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0cbfd3ff-073f-4383-a751-13815061dcb6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:34.987654 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0cbfd3ff-073f-4383-a751-13815061dcb6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=1828a302-02a8-45ee-b4e3-a006452b0dc2,source_replicaid=,source_volid=None,volume=Volume(6670b62f-dea7-4231-b7f1-c54b9a699852),volume_id=6670b62f-dea7-4231-b7f1-c54b9a699852,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:34.989553 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (db562ee0-939f-4363-b160-8e7a8d85f885) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:34.997196 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:15:35.012407 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 14, 55, 949565)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:35.012723 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:35.013094 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:35.013454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (6670b62f-dea7-4231-b7f1-c54b9a699852) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:35.021447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:32.651871 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:35.021447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:35.021447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:35.021447 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:32.651871] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:35.021983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:35.021983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:35.021983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:15:35.021983 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.32, total_capacity_gb: 19.77,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:35.018588 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:35.044780 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (db562ee0-939f-4363-b160-8e7a8d85f885) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:35.047113 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9df5e320-281c-42f8-9e6b-f7a93eb6278e tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (79f1a7bc-7264-4248-812b-a5139f0b00ec) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:36.719423 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-09a9d60c-3653-4528-8370-93f63dda05df tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:37.483834 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (31d66eba-202e-4c5a-9317-d65708b3857f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:37.486454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8286f1ba-cd54-4b84-a0fb-440fe39d9808) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:37.488897 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8286f1ba-cd54-4b84-a0fb-440fe39d9808) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2514cb45-cdd0-4ed4-ad32-552c353edfa1),volume_id=2514cb45-cdd0-4ed4-ad32-552c353edfa1,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:37.490843 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5900c967-7150-4d8f-9172-94231f5d9637) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:37.498247 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:15:37.518674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 36, 711626)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:37.518674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:37.518674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:37.518674 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2514cb45-cdd0-4ed4-ad32-552c353edfa1) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:36.711626 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:37.519085 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:15:37.520840 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:15:37.523213 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:37.525361 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:37.528260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:36.711626] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:37.529460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:37.529460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:37.529460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:15:37.529460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:37.518758 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:37.544785 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5900c967-7150-4d8f-9172-94231f5d9637) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:37.546829 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2ce719c6-150f-4333-8a10-fe7a6426818f tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (31d66eba-202e-4c5a-9317-d65708b3857f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:38.695758 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 36, 711626)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:38.696314 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:38.696764 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:38.696887 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:38.697275 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:37.518758 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:38.698434 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:38.699152 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:38.699537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:37.518758] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:38.699946 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:38.700299 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:38.700695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-50dfbbac-c5bd-4c23-80ab-0709adebe389 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:38.700344 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:40.017911 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (af64a242-5b63-4bf5-bd9b-733206d98968) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:40.020914 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (787de929-3008-4c63-b483-555ba9b29277) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:40.023585 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (787de929-3008-4c63-b483-555ba9b29277) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=878d30af-38d7-47f9-89d5-4271e52dcf20,source_replicaid=,source_volid=None,volume=Volume(1dc073eb-bb1d-4c06-b419-35008ffa2b1a),volume_id=1dc073eb-bb1d-4c06-b419-35008ffa2b1a,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:40.026731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e465fe43-215a-49e7-af76-54435a933685) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:15:40.034404 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:15:40.048911 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 36, 711626)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:15:40.049338 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:15:40.049710 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:40.050072 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1dc073eb-bb1d-4c06-b419-35008ffa2b1a) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:15:40.050470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:38.700344 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:15:40.051009 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:40.053177 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:15:40.055404 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:15:40.058037 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:15:40.060380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:40.060380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:15:40.060380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:38.700344] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:15:40.060380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:15:40.060380 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:15:40.060858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:15:40.060858 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.77,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:40.053976 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:15:40.080083 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e465fe43-215a-49e7-af76-54435a933685) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:15:40.081489 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b468cc67-17e1-4f5c-a939-7f8d1b6223a3 tempest-VolumesSnapshotTestJSON-970988873 None] Flow 'volume_create_scheduler' (af64a242-5b63-4bf5-bd9b-733206d98968) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:15:41.610509 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5cbbac3d-a414-4a6f-bc9a-cf07ef8b9e15 tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:43.838208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7d342842-fbfc-4a47-b489-c592704e6f73 tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:45.855740 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e0d04500-248b-407b-9de0-29a92b73f912 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:15:46.147299 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9beb78e2-3d6c-4c2e-9cd0-3a9a9de68f62 tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:47.992144 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8b0aef84-2dc3-46df-b869-b3388ef4cb9b tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:49.699415 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c753e37-1d9e-4dbf-8ae7-1feda063ff17 tempest-VolumesSnapshotTestJSON-970988873 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:15:55.992435 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4a2cd564-6b9c-44e3-a236-b9150124e575 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:16:01.011504 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Flow 'volume_create_scheduler' (1fdaa152-3010-4bc6-84d5-865b72be036b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:01.015236 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a7699643-1120-4d9f-8d66-9d83e47f206c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:01.018682 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a7699643-1120-4d9f-8d66-9d83e47f206c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(91367eb9-c68e-4706-b90d-3458b60402b3),volume_id=91367eb9-c68e-4706-b90d-3458b60402b3,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:01.021323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (954abf2c-f749-4787-a1a1-f3ccc476f55d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:01.029019 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:16:01.046536 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 55, 988021)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:16:01.046904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:16:01.047913 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:01.048274 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (91367eb9-c68e-4706-b90d-3458b60402b3) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:16:01.049016 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:55.988021 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:16:01.050854 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:16:01.053247 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:16:01.056293 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:16:01.059629 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:16:01.063613 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:01.063613 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:01.063613 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:15:55.988021] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:16:01.063613 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Weigher CapacityWeigher returned, weigher value is {max: 390.20000000000005, min: 390.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:16:01.064218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:16:01.064218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:16:01.064218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:01.054972 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:16:01.085439 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (954abf2c-f749-4787-a1a1-f3ccc476f55d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:01.087610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-afc5fd1c-a197-430b-94bc-8d4806467a32 tempest-ServersNegativeTestJSON-1448870888 None] Flow 'volume_create_scheduler' (1fdaa152-3010-4bc6-84d5-865b72be036b) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:03.206744 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Flow 'volume_create_scheduler' (9de2c112-920f-4cd8-902a-dbabdd695e9e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:03.209840 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4b5f07f3-e73b-4221-8659-86c43c890ae7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:03.212253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (4b5f07f3-e73b-4221-8659-86c43c890ae7) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a15b36c5-3431-4be0-a125-5122bad5b295),volume_id=a15b36c5-3431-4be0-a125-5122bad5b295,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:03.214133 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0eb0ccf0-63c4-4ef2-9dfc-329e9c78d1ca) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:03.222749 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:16:03.236156 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 55, 988021)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:16:03.236618 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:16:03.236899 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:03.237260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a15b36c5-3431-4be0-a125-5122bad5b295) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:16:03.237696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:01.054972 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:16:03.238793 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:16:03.240630 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:16:03.242610 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:16:03.244790 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:16:03.248104 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:03.248104 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:03.248104 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:01.054972] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:16:03.248104 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Weigher CapacityWeigher returned, weigher value is {max: 389.20000000000005, min: 389.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:16:03.248568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:16:03.248568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:16:03.248568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:03.241149 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:16:03.269770 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (0eb0ccf0-63c4-4ef2-9dfc-329e9c78d1ca) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:03.271818 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-624978d0-a6b6-4211-b857-1a859575314b tempest-ServersNegativeTestJSON-1448870888 None] Flow 'volume_create_scheduler' (9de2c112-920f-4cd8-902a-dbabdd695e9e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:33.515429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Flow 'volume_create_scheduler' (dedf34bc-a883-4908-85c7-f9f2538edd8d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:33.520789 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (35a09b5c-5dc1-4bb1-bdcf-cf924fadd664) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:33.523406 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (35a09b5c-5dc1-4bb1-bdcf-cf924fadd664) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(1cc1bd54-c208-4cec-ae92-52b6cfb3d6d8),volume_id=1cc1bd54-c208-4cec-ae92-52b6cfb3d6d8,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:33.525306 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b69ca92f-85aa-439f-8a5e-1d553d71d44d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:16:33.532870 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:16:33.551492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 15, 55, 988021)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:16:33.551886 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:16:33.552255 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:33.552802 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1cc1bd54-c208-4cec-ae92-52b6cfb3d6d8) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:16:33.553363 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:03.241149 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:16:33.554953 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:33.555528 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:16:33.556093 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:03.241149] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:16:33.556518 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Weigher CapacityWeigher returned, weigher value is {max: 388.20000000000005, min: 388.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:16:33.556876 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:16:33.557355 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:16:33.557732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.76,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:16:33.557380 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:16:33.589580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b69ca92f-85aa-439f-8a5e-1d553d71d44d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:16:33.591925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-1517ad3d-a6b4-4e6b-8cab-6aa9ee32b273 tempest-AttachVolumeShelveTestJSON-486918854 None] Flow 'volume_create_scheduler' (dedf34bc-a883-4908-85c7-f9f2538edd8d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:16:45.854342 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4875871e-dfa7-4ba5-a117-fa2c3e53970e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:16:55.933296 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fae28782-83fd-4b73-b1b2-41b1e0e2989a None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:20.333931 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8d1f4e79-333c-470a-ba78-eec707884f3e tempest-ServersNegativeTestJSON-1448870888 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:21.615911 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f1ab8bae-17b1-4e4a-b204-12e1664894ec tempest-ServersNegativeTestJSON-1448870888 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:31.320543 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (a0ced052-788c-4f6d-b6cc-51c35741bb6c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:31.323798 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (11174fcb-cd4d-458c-a938-77fb2ee5c513) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:31.326630 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (11174fcb-cd4d-458c-a938-77fb2ee5c513) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(010ee260-e923-47e3-a596-a5f6dda9560c),volume_id=010ee260-e923-47e3-a596-a5f6dda9560c,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:31.331031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8dda77f-dd1f-4ba5-9f0e-bdd723269b04) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:31.340246 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:17:31.359715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 17, 21, 606866)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:17:31.360467 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:17:31.361043 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:31.361622 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (010ee260-e923-47e3-a596-a5f6dda9560c) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:17:31.362520 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:21.606866 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:17:31.363057 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:17:31.365157 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:17:31.367480 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:17:31.369364 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:17:31.374153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:31.374153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:31.374153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.32, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:21.606866] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:17:31.374153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Weigher CapacityWeigher returned, weigher value is {max: 389.20000000000005, min: 389.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:17:31.374153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:17:31.374811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:17:31.374811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:31.366028 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:17:31.400286 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c8dda77f-dd1f-4ba5-9f0e-bdd723269b04) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:31.402684 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-15b2c008-ea06-4b57-98e3-300389a84c69 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (a0ced052-788c-4f6d-b6cc-51c35741bb6c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:32.736559 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (b9b60a35-5ea3-41e9-b013-30954ed1110a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:32.739028 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (206a5699-0ad7-4ae6-9ec0-0ca815f46702) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:32.741672 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (206a5699-0ad7-4ae6-9ec0-0ca815f46702) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a822d9b9-dcad-44c9-b976-b7d8915c4dcf),volume_id=a822d9b9-dcad-44c9-b976-b7d8915c4dcf,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:32.743814 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (25428364-7db9-4196-8b60-182d8aa812c4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:32.755284 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:17:32.770971 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 17, 21, 606866)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:17:32.771420 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:17:32.771787 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:32.772147 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (a822d9b9-dcad-44c9-b976-b7d8915c4dcf) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:17:32.772615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:31.366028 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:17:32.773661 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:17:32.776187 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:17:32.779096 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:17:32.783909 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:17:32.789184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:32.789184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:32.789184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.32, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:31.366028] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:17:32.789184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Weigher CapacityWeigher returned, weigher value is {max: 388.20000000000005, min: 388.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:17:32.789184 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:17:32.789986 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:17:32.789986 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:32.777009 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:17:32.805013 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (25428364-7db9-4196-8b60-182d8aa812c4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:32.807136 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b54a79a3-902b-4642-9a15-939e59b889c9 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (b9b60a35-5ea3-41e9-b013-30954ed1110a) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:34.092185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (c4facf2b-bda6-41f2-9f7d-48bec8fe50a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:34.095111 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1daa0c3e-9225-4e49-8819-45c222f48f7e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:34.097753 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1daa0c3e-9225-4e49-8819-45c222f48f7e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4ffbde95-6f78-48da-a0eb-1d590a860308),volume_id=4ffbde95-6f78-48da-a0eb-1d590a860308,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:34.099657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (aa93b6aa-be3d-4e49-98dd-ea855f56b074) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:17:34.107922 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:17:34.122182 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 17, 21, 606866)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:17:34.122627 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:17:34.122982 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:34.123340 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4ffbde95-6f78-48da-a0eb-1d590a860308) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:17:34.123732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:32.777009 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:17:34.124853 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:17:34.126835 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:17:34.128915 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:17:34.131686 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:17:34.135153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:34.135153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:17:34.135153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.32, total_capacity_gb: 19.76,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:32.777009] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:17:34.135153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Weigher CapacityWeigher returned, weigher value is {max: 387.20000000000005, min: 387.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:17:34.135153 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:17:34.136002 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:17:34.136002 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.32, total_capacity_gb: 19.76,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:17:34.127229 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:17:34.156048 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (aa93b6aa-be3d-4e49-98dd-ea855f56b074) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:17:34.158217 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-21647fd0-3277-40d4-81c3-5b52deba71f7 tempest-VolumesTestJSON-750256982 None] Flow 'volume_create_scheduler' (c4facf2b-bda6-41f2-9f7d-48bec8fe50a6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:17:36.782670 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-677e5fa0-940f-4c9e-a976-fba302086038 tempest-VolumesTestJSON-750256982 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:37.997469 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f5f360f8-ba75-4df5-9256-9f952d5d941c tempest-VolumesTestJSON-750256982 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:39.264796 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4b8abc63-9905-4cc1-9501-b5f1863e4156 tempest-VolumesTestJSON-750256982 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:17:45.860944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c5d80fb5-64b8-42ae-9d37-0d5d1756e9aa None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:17:55.974895 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6b2c900a-3e98-488a-b175-31638742a915 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:18:45.870000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f7fb679-9264-40e5-8b44-411071be5268 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:18:55.996924 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e44c9c96-93f7-4b69-8b23-b9b22e766f67 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:18:56.452910 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-68fc88a6-209f-4499-8210-711034ba0e6b tempest-AttachVolumeShelveTestJSON-486918854 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:18:57.474698 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3ef57a66-bb4d-4364-a095-c920a954ac91 tempest-AttachVolumeShelveTestJSON-486918854 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:19:07.793977 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (f76d99ab-bf96-4ca6-bbeb-e4a8cd52b483) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:07.797572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (081a1a76-1d9e-4cbb-a837-1d3dba1cac72) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:07.800900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (081a1a76-1d9e-4cbb-a837-1d3dba1cac72) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4930315f-a35d-47e5-816e-622618a3ea5b),volume_id=4930315f-a35d-47e5-816e-622618a3ea5b,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:07.802904 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e94e8bb8-79d5-4747-9df1-adeb56e1d624) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:07.818091 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:19:07.848658 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 18, 57, 466674)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:07.849241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:07.849743 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:07.850329 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4930315f-a35d-47e5-816e-622618a3ea5b) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:07.850955 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.76,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:18:57.466674 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:07.853420 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:07.853931 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:07.854436 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.76,allocated_capacity_gb: 4, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 4,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:18:57.466674] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:07.855763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 391.20000000000005, min: 391.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:07.859101 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:07.859807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:19:07.860274 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:07.859840 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:07.902695 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e94e8bb8-79d5-4747-9df1-adeb56e1d624) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:07.910145 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2166294b-2923-468a-b584-0d7b181a6dfa tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (f76d99ab-bf96-4ca6-bbeb-e4a8cd52b483) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:10.356683 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 18, 57, 466674)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:10.357219 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:10.357449 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:10.360823 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:10.361281 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:07.859840 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.4, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:10.361859 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:19:10.363329 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:19:10.364910 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:10.366261 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:10.371537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:10.371537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:10.371537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:07.859840] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:10.371537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 390.20000000000005, min: 390.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:10.371537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:10.372284 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ba34b4a5-518a-4437-be39-214f924aeea3 tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:10.364953 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:13.987384 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (59f2321a-e5b6-4807-8cbe-4e69a0b03471) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:13.992357 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (45a692b9-ddf6-4c11-ad0a-b06ad04db4a6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:13.997503 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (45a692b9-ddf6-4c11-ad0a-b06ad04db4a6) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=0ab36a77-45ff-4685-a6ce-2ffb9c4e46bb,source_replicaid=,source_volid=None,volume=Volume(5af74677-0ff1-495e-8304-d139df858d27),volume_id=5af74677-0ff1-495e-8304-d139df858d27,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:13.999676 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c4061d43-c984-4c51-a5d1-78e3851feb10) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:14.007751 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:19:14.033877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 18, 57, 466674)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:14.034280 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:14.034678 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:14.035034 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5af74677-0ff1-495e-8304-d139df858d27) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:14.035642 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:10.364953 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:14.036742 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:14.038240 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:19:14.040862 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:14.046065 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:10.364953] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:14.050610 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 389.20000000000005, min: 389.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:14.051625 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:14.051625 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:19:14.051625 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:14.039368 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:14.070703 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (c4061d43-c984-4c51-a5d1-78e3851feb10) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:14.073118 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-aa720099-3b58-48ee-a865-6f62cb2da12c req-85ce01e5-0691-4fd0-a00a-cb195d12cce1 tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (59f2321a-e5b6-4807-8cbe-4e69a0b03471) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:20.022861 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (61bd8b5e-24cc-47d2-8035-f62bb023997f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:20.029560 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8c251d73-d5c4-4e5b-9afa-925033598c4e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:20.032363 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8c251d73-d5c4-4e5b-9afa-925033598c4e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(08e15870-5cd2-4d69-916b-b7c7e907ddc1),volume_id=08e15870-5cd2-4d69-916b-b7c7e907ddc1,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:20.035440 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e697283a-44f7-434b-9b9f-b7569eb990dc) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:20.048329 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:19:20.066704 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 4, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 18, 57, 466674)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:20.067807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:20.068501 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:20.069179 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (08e15870-5cd2-4d69-916b-b7c7e907ddc1) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:20.069596 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:14.039368 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:20.071049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:20.071609 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:20.072131 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.31, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:14.039368] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:20.072791 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Weigher CapacityWeigher returned, weigher value is {max: 388.20000000000005, min: 388.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:20.073853 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:20.074494 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:19:20.075192 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.309999999999999, total_capacity_gb: 19.76,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:20.074706 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:20.106918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e697283a-44f7-434b-9b9f-b7569eb990dc) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:20.108886 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-424ca4aa-13fa-415c-bfb4-730285a8ea24 tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (61bd8b5e-24cc-47d2-8035-f62bb023997f) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:34.277269 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-cddb15c7-7a66-4112-901a-4ad5a0eacb58 req-466b5b8e-6b96-4cae-bdd5-c9c536b10aa2 tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:19:38.116702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-42dbb2f8-a7b9-4499-b251-e493e15a0bfc tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:19:39.517211 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (b42c63d0-27ea-4982-adee-45edd5e225a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:39.520432 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2d1bf314-db1d-4241-a5be-bcd80fcf723d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:39.522944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2d1bf314-db1d-4241-a5be-bcd80fcf723d) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(de4f48ae-e75d-442f-bb31-a458213e7d85),volume_id=de4f48ae-e75d-442f-bb31-a458213e7d85,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:39.524866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7d94c2be-65a4-4640-b37c-51fdfec46aec) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:39.533069 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:19:39.547174 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 38, 111118)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:39.547715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:39.548243 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:39.548761 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (de4f48ae-e75d-442f-bb31-a458213e7d85) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:39.549593 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:38.111118 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:39.551336 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:19:39.553425 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:39.555202 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:39.556969 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:39.558950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:39.558950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:39.558950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.31, total_capacity_gb: 19.76,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:38.111118] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:39.558950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 390.20000000000005, min: 390.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:39.558950 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:39.559303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:19:39.559303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:39.553758 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:39.584387 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7d94c2be-65a4-4640-b37c-51fdfec46aec) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:39.587245 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-844b73d3-461e-4f2b-a589-286e6617a9de tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (b42c63d0-27ea-4982-adee-45edd5e225a4) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:45.870051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-64b9f047-f35d-4ac0-9c4e-0aed96c58456 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:19:54.731705 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (a5331f28-8257-42c4-84dc-cd042736ec82) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:54.738620 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a3f8b701-fd78-4282-b44f-077e77d44d5c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:54.743337 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (a3f8b701-fd78-4282-b44f-077e77d44d5c) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(e1be6716-6919-4e02-b766-ee81bbe16518),volume_id=e1be6716-6919-4e02-b766-ee81bbe16518,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:54.747063 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d1338da0-8881-444b-aee0-66bfbfea836b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:19:54.754655 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:19:54.770177 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.31, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 38, 111118)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:54.770604 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:54.770973 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:54.771330 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (e1be6716-6919-4e02-b766-ee81bbe16518) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:54.771728 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:39.553758 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:19:54.772760 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:19:54.774160 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:19:54.776446 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:19:54.778045 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:19:54.779419 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:19:54.779419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:54.779419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:54.779419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.31, total_capacity_gb: 19.76,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:39.553758] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:54.779419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Weigher CapacityWeigher returned, weigher value is {max: 389.20000000000005, min: 389.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:54.779747 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:54.779747 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:19:54.779747 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.31, total_capacity_gb: 19.76,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:54.775668 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:19:54.805691 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d1338da0-8881-444b-aee0-66bfbfea836b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:19:54.807733 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-067c2cdd-d8e4-4ec8-b86b-980865ecfed5 tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (a5331f28-8257-42c4-84dc-cd042736ec82) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:19:55.935612 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d12990cb-27a0-4436-bbb7-1d17bfea4b9b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:19:56.992493 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 55, 930162)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:19:56.993968 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:19:56.994507 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:56.994856 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:19:56.995285 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:55.930162 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:19:56.996522 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:56.996907 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:19:56.997303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.33, total_capacity_gb: 19.77,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:55.930162] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:19:57.004098 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 388.4, min: 388.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:19:57.006821 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:19:57.007261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-9845f988-60b3-4856-9a47-80ddf8de86d3 req-6f636b26-acd1-4468-b6b3-7dbc0e799f2e tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:57.006876 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:20:01.295067 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (5008f327-15ee-4d57-a64a-2644c19228cd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:01.301577 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (15fbd13d-d648-4632-ac43-ea36ec750195) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:01.310780 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (15fbd13d-d648-4632-ac43-ea36ec750195) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=a3ed6062-9235-41cc-8be3-469449fbeed3,source_replicaid=,source_volid=None,volume=Volume(0185cfe3-743b-4efa-a9d0-db104d5391c4),volume_id=0185cfe3-743b-4efa-a9d0-db104d5391c4,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:01.312490 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ef7369c5-6052-465e-9339-d0d6492cfab8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:01.327096 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:20:01.346868 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 55, 930162)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:20:01.354575 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:20:01.355458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:01.355956 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0185cfe3-743b-4efa-a9d0-db104d5391c4) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:20:01.356574 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:57.006876 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:20:01.358197 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:20:01.360116 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:20:01.362444 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:20:01.364950 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.33, total_capacity_gb: 19.77,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:19:57.006876] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:20:01.367172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:20:01.368312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:20:01.368312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:20:01.368312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:01.361231 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:20:01.405261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (ef7369c5-6052-465e-9339-d0d6492cfab8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:01.407599 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [req-39289fa6-8bbe-4e3e-a29b-01a65eb60b50 req-ba3eb342-c132-406e-84e7-4aa94486584e tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (5008f327-15ee-4d57-a64a-2644c19228cd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:08.157907 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Flow 'volume_create_scheduler' (78819d37-ca75-423d-a5c5-20f3ddc3bfd6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:08.161127 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (87d3dca0-7cb1-43b8-ac3e-d6fcbec5fcdd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:08.163578 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (87d3dca0-7cb1-43b8-ac3e-d6fcbec5fcdd) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(0d8ffd7d-a4e5-4a95-9fa8-5c805d55ed66),volume_id=0d8ffd7d-a4e5-4a95-9fa8-5c805d55ed66,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:08.165977 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6baa2be1-ac60-48f2-875c-b0e76e28ed06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:08.173603 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:20:08.188006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 55, 930162)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:20:08.188368 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:20:08.188748 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:08.189110 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (0d8ffd7d-a4e5-4a95-9fa8-5c805d55ed66) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:20:08.189514 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:01.361231 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:08.190572 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:20:08.192269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:20:08.193669 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:20:08.196270 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:20:08.198194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:08.198194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:08.198194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.33, total_capacity_gb: 19.77,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:01.361231] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:20:08.198194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:20:08.198194 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:20:08.198653 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:20:08.198653 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:08.192907 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:20:08.220042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (6baa2be1-ac60-48f2-875c-b0e76e28ed06) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:08.222433 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-94aca4e9-8f36-4ff1-b1a3-f92a4ad3a8c6 tempest-VolumesActionsTest-1114221274 None] Flow 'volume_create_scheduler' (78819d37-ca75-423d-a5c5-20f3ddc3bfd6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:13.907579 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (1e398fc5-db1b-40ff-97c0-a43ee7656aad) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:13.910397 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (eaa8dcaf-3098-452c-9f72-446331a66ba0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:13.912845 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (eaa8dcaf-3098-452c-9f72-446331a66ba0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7d290b0a-05d8-4750-aef6-9132080bc747),volume_id=7d290b0a-05d8-4750-aef6-9132080bc747,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:13.915290 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e30f4cf9-6ba8-481a-8874-77222ba817f4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:20:13.922413 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:20:13.936630 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.33, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 19, 55, 930162)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:20:13.937069 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:20:13.937437 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:13.937795 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (7d290b0a-05d8-4750-aef6-9132080bc747) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:20:13.938190 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:08.192907 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:20:13.939603 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:13.940149 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:20:13.940519 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.33, total_capacity_gb: 19.77,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:08.192907] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:20:13.940929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:20:13.941277 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:20:13.941746 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:20:13.942122 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.329999999999998, total_capacity_gb: 19.77,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:13.941774 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:20:13.967892 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (e30f4cf9-6ba8-481a-8874-77222ba817f4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:20:13.969763 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-fe43f5ac-7378-45e5-a9ed-c1500152c5ba tempest-AttachVolumeNegativeTest-1465826611 None] Flow 'volume_create_scheduler' (1e398fc5-db1b-40ff-97c0-a43ee7656aad) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:20:18.974119 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-2137987b-0ad2-45c7-b187-1e72774f17b0 req-b2f56b7c-0c22-4711-8221-b317cd94ff14 tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:20:20.538046 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-13feda07-70ed-41ee-b8a7-def48bf2f70e tempest-AttachVolumeNegativeTest-1465826611 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.77, 'free_capacity_gb': 19.32, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 8, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:20:25.547718 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4345e1ea-c08c-4421-a9b2-44002d9b7168 tempest-AttachVolumeNegativeTest-1465826611 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.78, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:20:34.136772 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-45251a87-ecb4-4a6b-84c6-d2b079427e01 tempest-AttachVolumeNegativeTest-1465826611 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.76, 'free_capacity_gb': 19.28, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:20:45.864716 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-aa4e5679-7f14-40f3-9420-2b1fe0a2046b None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:20:55.950419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-539c3970-5a4a-45cc-90a4-7e5c1d5ea132 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 6, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:20:56.197177 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-9639aed3-df4b-4ba4-b7b2-ccf488e1be7b req-d1eab0f9-0638-4e11-b775-6d6c53b5d9fc tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:21:01.889399 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (cf13af1f-a3a2-4866-a936-42261bd13535) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:21:01.893896 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d743cd75-2b6a-4d6a-b6b7-a43ef18b4c33) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:21:01.896550 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (d743cd75-2b6a-4d6a-b6b7-a43ef18b4c33) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2f8227b8-bd5e-456e-866f-fd0bf6c93d27),volume_id=2f8227b8-bd5e-456e-866f-fd0bf6c93d27,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:21:01.898702 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (44e7e09d-5d4f-48ea-9f23-0e7fda69fea7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:21:01.907866 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:21:01.967171 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.74, 'free_capacity_gb': 19.23, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 20, 56, 192365)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:21:01.967470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:21:01.967596 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:01.968339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2f8227b8-bd5e-456e-866f-fd0bf6c93d27) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:21:01.969113 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.74,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:56.192365 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:21:01.970020 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:21:01.972200 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:01.974354 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:21:01.976090 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:21:01.977876 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:21:01.980193 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:01.980786 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:01.981334 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.23, total_capacity_gb: 19.74,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:20:56.192365] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:21:01.982029 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 389.79999999999995, min: 389.79999999999995} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:21:01.982550 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:21:01.983172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:21:01.983714 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.23, total_capacity_gb: 19.74,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:01.983361 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:21:02.032958 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (44e7e09d-5d4f-48ea-9f23-0e7fda69fea7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:21:02.037420 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8fe3d5d4-48c7-4615-aa59-d2f345c4d350 tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (cf13af1f-a3a2-4866-a936-42261bd13535) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:21:09.406883 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dec485cb-4a59-44f7-a8ff-40626288bf67 tempest-VolumesActionsTest-1114221274 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:21:12.117781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Flow 'volume_create_scheduler' (79168267-c2c0-475f-9bd7-cb5bad6fcbb0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:21:12.125754 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1bc90980-812e-447c-8d4b-42f58b3b8ecf) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:21:12.140344 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1bc90980-812e-447c-8d4b-42f58b3b8ecf) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(7b8fecfb-8a31-476e-9a25-646d616ef209),volume_id=7b8fecfb-8a31-476e-9a25-646d616ef209,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:21:12.145325 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (783048f8-b838-4b2c-b4ca-8105563efa15) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:21:12.167397 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:21:12.183076 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 9, 402361)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:21:12.183571 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:21:12.183996 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:12.184955 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (7b8fecfb-8a31-476e-9a25-646d616ef209) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:21:12.185417 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.3, total_capacity_gb: 19.75,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:09.402361 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:21:12.186535 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:12.188461 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:21:12.189885 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:21:12.191493 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.3, total_capacity_gb: 19.75,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:09.402361] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:21:12.193611 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Weigher CapacityWeigher returned, weigher value is {max: 390.0, min: 390.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:21:12.194120 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:21:12.194120 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:21:12.194120 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.3, total_capacity_gb: 19.75,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:12.193647 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:21:12.223827 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (783048f8-b838-4b2c-b4ca-8105563efa15) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:21:12.226803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-562c2812-58d7-48bd-9be9-45c96bf895ab tempest-VolumesSnapshotListTestJSON-1658940931 None] Flow 'volume_create_scheduler' (79168267-c2c0-475f-9bd7-cb5bad6fcbb0) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:21:13.355915 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 9, 402361)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:21:13.356504 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:21:13.356671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:13.357031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:21:13.357416 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.3, total_capacity_gb: 19.75,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:12.193647 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:21:13.358655 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:13.359013 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:13.359394 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.3, total_capacity_gb: 19.75,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:12.193647] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:21:13.359793 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Weigher CapacityWeigher returned, weigher value is {max: 389.0, min: 389.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:21:13.360264 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:21:13.360665 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d40a4a7b-4dbc-40ad-b90f-787ea915bc63 tempest-VolumesSnapshotListTestJSON-1658940931 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.3, total_capacity_gb: 19.75,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:13.360314 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:21:14.542663 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 9, 402361)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:21:14.543290 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:21:14.543419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:14.543776 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:21:14.544227 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.3, total_capacity_gb: 19.75,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:13.360314 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:21:14.545041 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 386.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:21:14.546717 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:21:14.548317 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 366.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:21:14.551554 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.3, total_capacity_gb: 19.75,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:13.360314] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:21:14.553835 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Weigher CapacityWeigher returned, weigher value is {max: 388.0, min: 388.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:21:14.554306 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:21:14.554306 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b0e3e2da-4c94-45ab-9fef-5bf9f2e9dfe9 tempest-VolumesSnapshotListTestJSON-1658940931 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.3, total_capacity_gb: 19.75,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:14.547044 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:21:15.724371 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 9, 402361)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:21:15.724962 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:21:15.725352 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:15.725807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:21:15.726256 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.3, total_capacity_gb: 19.75,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:14.547044 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:21:15.728028 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:15.728493 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:21:15.728935 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.3, total_capacity_gb: 19.75,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:14.547044] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:21:15.729546 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Weigher CapacityWeigher returned, weigher value is {max: 387.0, min: 387.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:21:15.729965 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:21:15.730438 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c7fa8f7-8282-4501-a27e-532ab1e2f8fa tempest-VolumesSnapshotListTestJSON-1658940931 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 15.3, total_capacity_gb: 19.75,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:15.730013 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:21:21.326582 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a616b714-04c8-4d79-bef0-63db3f39096d tempest-VolumesSnapshotListTestJSON-1658940931 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.75, 'free_capacity_gb': 19.3, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:21:45.859044 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5dbeb481-33f1-4c83-b1fc-b429a9acbc39 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:21:55.978439 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2b263cb7-0cb2-4d51-a300-2dd649d87c6d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.73, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:02.730563 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.73, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 55, 973611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:02.731215 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:02.731530 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.731905 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:02.732258 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.732629 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.732986 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.73,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:55.973611] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:22:02.733671 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 389.6, min: 389.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:02.734032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:02.742298 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.73, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 55, 973611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:02.742815 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:02.743592 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.744103 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:02.744639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.745655 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:02.746171 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.73,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:55.973611] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:22:02.747343 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 389.6, min: 389.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:02.747765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:02.748134 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-80f2e5d2-33d8-4a2c-91f8-eca6b0bf72d8 tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:22:04.094647 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (245fcd3c-82aa-49a1-84e4-d5c83e0247ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:04.097384 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8f3c7df5-5b12-4b73-9f5e-391e3435d815) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:04.099900 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8f3c7df5-5b12-4b73-9f5e-391e3435d815) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=1c48332f-400d-46d5-b557-dcaff8fd4e30,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(96b4ccf1-7644-43ea-ac5a-b530752a843c),volume_id=96b4ccf1-7644-43ea-ac5a-b530752a843c,volume_properties=VolumeProperties,volume_type=VolumeType(495b6ccc-ccee-4369-af43-0e59f0192c30))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:04.102642 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4b6b6e42-a708-4326-a278-7b674ccb58b0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:04.113241 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:04.127492 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.73, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 55, 973611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:04.127910 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:04.128277 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:04.128640 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (96b4ccf1-7644-43ea-ac5a-b530752a843c) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:04.129043 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.73,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:55.973611 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 346.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:04.130080 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:04.132122 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 326.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:04.134299 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:04.137269 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:04.140339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:04.140339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:04.140797 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.25, total_capacity_gb: 19.73,allocated_capacity_gb: 5, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 5,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:21:55.973611] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:04.141836 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 389.6, min: 389.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:04.142206 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:04.142781 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:04.143151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.73,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:04.142785 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:04.172807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4b6b6e42-a708-4326-a278-7b674ccb58b0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:04.175572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c19f4f24-ef5b-40b6-b249-33c9f65d6c28 tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (245fcd3c-82aa-49a1-84e4-d5c83e0247ea) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:06.740192 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.73, 'free_capacity_gb': 19.25, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 5, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 21, 55, 973611)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:06.741185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:06.741185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:06.741406 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:06.741770 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.73,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:04.142785 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:06.743061 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:06.743426 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:06.743964 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.25, total_capacity_gb: 19.73,allocated_capacity_gb: 6, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 6,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:04.142785] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:06.744388 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 388.6, min: 388.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:06.745344 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:06.745706 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8760010c-4b1d-4b63-8dfd-2b43a498188e tempest-GroupSnapshotsTest-672204617 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.25, total_capacity_gb: 19.73,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:06.745335 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:09.337220 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b39b69ef-47b8-41dd-988a-522f86154d0a tempest-GroupSnapshotsTest-672204617 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:12.030948 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b8752ae0-dde4-4540-9319-888907024faa tempest-GroupSnapshotsTest-672204617 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:12.626811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 12, 20445)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:12.627630 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:12.628251 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.628681 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:12.629503 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.630291 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.631263 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:12.020445] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:22:12.631922 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:12.632551 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:12.641706 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 12, 20445)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:12.642364 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:12.642971 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.643551 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:12.644152 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.644743 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:12.645334 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:12.020445] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:22:12.645988 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:12.647186 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:12.647764 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-992a42da-2a61-4047-bf4d-e9a9e348d099 tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:22:14.020615 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (a522e336-c994-4c46-81ae-768a64abbb8c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:14.024972 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (84f8bc4c-986f-40fa-8b17-ba7f949535ff) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:14.030042 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (84f8bc4c-986f-40fa-8b17-ba7f949535ff) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=6fb12971-a17d-4a3e-b869-2957bea47f28,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(64caacd8-da49-4ddc-8609-fad907d00a5b),volume_id=64caacd8-da49-4ddc-8609-fad907d00a5b,volume_properties=VolumeProperties,volume_type=VolumeType(c399c90a-3486-451b-8403-7ef8b907eb6b))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:14.032945 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4f87fe7b-ed54-4e37-9931-da70a3b0d08c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:14.048636 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:14.064657 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 12, 20445)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:14.065537 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:14.065999 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:14.066453 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (64caacd8-da49-4ddc-8609-fad907d00a5b) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:14.066972 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:12.020445 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:14.068608 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 385.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:14.070501 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 444, in validate_host_capacity Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: backend_state = self.driver.backend_passes_filters( Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:14.073360 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 365.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:14.075221 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:14.077419 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:14.077419 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:14.077419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:14.077419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:14.077419 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 7, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 7,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:12.020445] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:14.078613 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 387.4, min: 387.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:14.079208 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:14.079777 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:14.080368 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.72,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:14.080011 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:14.119099 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4f87fe7b-ed54-4e37-9931-da70a3b0d08c) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:14.121944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-c85c1dc5-0ba4-4d39-a6d5-107403c4d431 tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (a522e336-c994-4c46-81ae-768a64abbb8c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:15.645914 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (ff88b072-9331-4681-937e-24c93c12147c) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:15.658154 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2886225c-d255-49cd-bb39-1de14b5c09e0) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:15.661749 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (2886225c-d255-49cd-bb39-1de14b5c09e0) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=6fb12971-a17d-4a3e-b869-2957bea47f28,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(bd6e4d88-9914-447e-b8b5-4992d4f6c4ba),volume_id=bd6e4d88-9914-447e-b8b5-4992d4f6c4ba,volume_properties=VolumeProperties,volume_type=VolumeType(c399c90a-3486-451b-8403-7ef8b907eb6b))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:15.667925 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (40ebf454-ef5a-4ce1-b498-3332144e3030) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:15.694895 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:15.730045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 7, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 12, 20445)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:15.732312 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:15.735132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:15.735749 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (bd6e4d88-9914-447e-b8b5-4992d4f6c4ba) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:15.736221 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.72,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:14.080011 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:15.737336 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:15.739303 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:15.741105 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:15.743638 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.72,allocated_capacity_gb: 8, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 8,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:14.080011] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 386.4, min: 386.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:15.747052 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:15.747956 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:15.749480 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.2, total_capacity_gb: 19.72,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:15.747904 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:15.801744 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (40ebf454-ef5a-4ce1-b498-3332144e3030) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:15.804002 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-b699ebdc-4a5c-4c94-8a06-3e89624195af tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (ff88b072-9331-4681-937e-24c93c12147c) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:27.440487 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9ae8c638-2863-44f8-8070-915c580afb78 tempest-GroupSnapshotsTest-672204617 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:27.797502 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 27, 435962)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:27.797918 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:27.798282 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.798670 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:27.799018 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.799399 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.799772 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:27.435962] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:22:27.800169 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:27.800700 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:27.808224 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 27, 435962)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:27.808627 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:27.808993 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.809347 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:27.809689 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.810041 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:27.810410 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:27.435962] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:22:27.810844 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:27.811607 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:27.811968 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-88ed0aa0-5e4a-4e73-8902-b7b1f8aff478 tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:22:29.117711 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (e2b5fa85-3cc3-405e-9a7d-c40b43f1238d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:29.120709 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c617b452-f5be-4932-ba1f-4abc038fd3a4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:29.123365 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c617b452-f5be-4932-ba1f-4abc038fd3a4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=4eb1fdb9-d266-4a21-9ca4-6f53d3970d34,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(390d480b-256a-44aa-863d-7ca052f6bd1f),volume_id=390d480b-256a-44aa-863d-7ca052f6bd1f,volume_properties=VolumeProperties,volume_type=VolumeType(71b2053a-f696-4dd0-99f1-7c85a6206c31))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:29.125248 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f453459-ca88-4659-b927-ffe1f1a9c2a9) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:29.132469 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:29.146145 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.72, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 9, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 27, 435962)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:29.146974 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:29.147340 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:29.147696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (390d480b-256a-44aa-863d-7ca052f6bd1f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:29.148097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:27.435962 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:29.149153 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:29.151216 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:29.153643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:29.155492 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:29.157303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:29.157303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:29.157303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.72,allocated_capacity_gb: 9, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 9,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:27.435962] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:29.157303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Weigher CapacityWeigher returned, weigher value is {max: 385.4, min: 385.4} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:29.157303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:29.157680 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:29.157680 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.72,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:29.151600 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:29.190403 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (7f453459-ca88-4659-b927-ffe1f1a9c2a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:29.192486 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2689a755-7517-4974-82d0-1abfbd39e65f tempest-GroupSnapshotsTest-672204617 None] Flow 'volume_create_scheduler' (e2b5fa85-3cc3-405e-9a7d-c40b43f1238d) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:33.408309 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b5b79d63-e3b2-4833-95fd-ad29de995680 tempest-GroupSnapshotsTest-672204617 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:37.289229 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 33, 403760)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:37.289934 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:37.290251 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.290358 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:37.290664 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.291319 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.291693 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:33.403760] {{(pid=133283) _get_weighted_candidates_by_group_type /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:540}} Dec 11 21:22:37.292088 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Weigher CapacityWeigher returned, weigher value is {max: 384.20000000000005, min: 384.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:37.292429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:37.298253 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 33, 403760)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:37.298693 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:37.299049 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.299397 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Checking if host ncdv-master-1@ceph#ceph can create a 0 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:37.299734 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.300079 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:37.300439 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:33.403760] {{(pid=133283) _get_weighted_candidates_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:448}} Dec 11 21:22:37.300817 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Weigher CapacityWeigher returned, weigher value is {max: 384.20000000000005, min: 384.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:37.301177 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:37.301526 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-deac762c-0347-4904-a9b5-1655645216a6 tempest-GroupsV314Test-653952660 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend_generic_group /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:615}} Dec 11 21:22:38.603408 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Flow 'volume_create_scheduler' (d3d20688-bf71-4cba-8093-14cf7067be00) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:38.611151 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0b988400-ee3e-4905-8dbc-3cd16045659a) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:38.613688 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (0b988400-ee3e-4905-8dbc-3cd16045659a) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=cda322c2-a43b-4538-9cde-63fd0d9bea27,image_id=None,operation=,resource_backend='ncdv-master-1@ceph',snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c7b658d1-2124-4791-a3c0-7660dd3f1c8f),volume_id=c7b658d1-2124-4791-a3c0-7660dd3f1c8f,volume_properties=VolumeProperties,volume_type=VolumeType(a76b2bea-518b-4ca6-a8c6-308e3027d185))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:38.615662 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5494ea90-c97e-416f-a390-5f7209b9e213) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:38.622706 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:38.636572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 33, 403760)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:38.636989 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:38.637360 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:38.637721 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c7b658d1-2124-4791-a3c0-7660dd3f1c8f) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:38.638122 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:33.403760 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:38.639544 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:38.639881 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:38.640262 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 10, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 10,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:33.403760] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:38.640676 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Weigher CapacityWeigher returned, weigher value is {max: 384.20000000000005, min: 384.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:38.641021 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:38.641383 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:38.641878 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:38.641411 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:38.668077 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (5494ea90-c97e-416f-a390-5f7209b9e213) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:38.670155 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9311c1b7-670e-4f07-a95e-2532bf67cba3 tempest-GroupsV314Test-653952660 None] Flow 'volume_create_scheduler' (d3d20688-bf71-4cba-8093-14cf7067be00) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:39.842715 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 10, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 33, 403760)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:39.843418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:39.843949 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:39.844362 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:39.844752 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:38.641411 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:39.845825 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:39.848266 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:39.850116 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:39.852038 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:39.854696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:39.854696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:39.854696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:38.641411] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:39.854696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Weigher CapacityWeigher returned, weigher value is {max: 383.20000000000005, min: 383.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:39.854696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:39.855071 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f3aa3c70-2b47-4173-8d4f-fd05d0bad858 tempest-GroupsV314Test-653952660 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.2, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:39.847697 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:42.557449 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2c896639-7d47-4633-8d46-63b1e96f3f57 tempest-GroupsV314Test-653952660 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:43.243460 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1264655c-de30-4750-b798-498e4c75f7bd tempest-GroupsV314Test-653952660 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:45.856442 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-abbe6063-b1ed-4129-9548-9a04ff793135 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:22:46.480047 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e1378b9a-3187-4330-ae11-f27e2f7dd9d8 tempest-VolumePoolsAdminTestsJSON-2134227524 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 43, 238014)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:46.510946 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0aff73c4-34b3-4814-9afb-1bfa363d0f38 tempest-VolumePoolsAdminTestsJSON-2134227524 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 43, 238014)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:52.131625 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (42599547-a20f-488a-9a15-a29cd3e5cfdd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:52.135196 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1b08bcad-6389-40ea-817d-0c499c975561) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:52.137572 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1b08bcad-6389-40ea-817d-0c499c975561) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(2ed1e77e-3c7e-4007-a18c-fee17a8143b9),volume_id=2ed1e77e-3c7e-4007-a18c-fee17a8143b9,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:52.139493 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (78415917-a5cb-4e19-8c8b-29149bbe4c9b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:52.146231 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:52.160642 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 43, 238014)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:52.161102 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:52.161514 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:52.161862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (2ed1e77e-3c7e-4007-a18c-fee17a8143b9) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:52.162362 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:43.238014 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:52.163764 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:52.164155 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:52.164562 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.2, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:43.238014] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:52.164984 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Weigher CapacityWeigher returned, weigher value is {max: 382.20000000000005, min: 382.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:52.165332 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:52.165803 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:52.166275 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:52.165829 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:52.191326 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (78415917-a5cb-4e19-8c8b-29149bbe4c9b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:52.193259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3de8ba74-15b4-49fc-8c0e-ab5761136d8a tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (42599547-a20f-488a-9a15-a29cd3e5cfdd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:53.526790 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (94d341f2-93b8-4436-97c6-288259124e87) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:53.529082 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (616bebed-baa5-4c2b-99bf-0248eec31844) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:53.531527 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (616bebed-baa5-4c2b-99bf-0248eec31844) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(50f6151b-756e-4f2e-a146-5f6d058a63dd),volume_id=50f6151b-756e-4f2e-a146-5f6d058a63dd,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:53.533916 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d8c56f4b-b381-42bb-8dc9-f31082948e35) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:53.541937 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:53.555337 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 43, 238014)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:53.555732 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:53.556096 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:53.556457 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (50f6151b-756e-4f2e-a146-5f6d058a63dd) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:53.556854 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:52.165829 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 444, in validate_host_capacity Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: backend_state = self.driver.backend_passes_filters( Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:53.557365 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:53.559501 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:53.561727 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 384.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:53.563636 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.2, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:52.165829] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:53.565576 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Weigher CapacityWeigher returned, weigher value is {max: 381.20000000000005, min: 381.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:53.566051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:53.566051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:53.566051 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.2, total_capacity_gb: 19.71,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:53.560267 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:53.585809 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d8c56f4b-b381-42bb-8dc9-f31082948e35) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:53.588636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-2f1e3aba-b76d-4c19-aa99-2d6839b462e0 tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (94d341f2-93b8-4436-97c6-288259124e87) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:54.873181 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (96fa9ee7-9de8-4d11-9bdb-cc18d77b93bd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:54.875960 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1ef9e773-22f2-4837-b0f6-67f2968a62ea) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:54.878463 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (1ef9e773-22f2-4837-b0f6-67f2968a62ea) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(916e2d16-6f68-4ffd-b4f6-b5d17c716574),volume_id=916e2d16-6f68-4ffd-b4f6-b5d17c716574,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:54.880616 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (beaf0ca6-e02c-4141-8dd0-14f1c5e61921) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:54.889056 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:54.909626 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.2, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 43, 238014)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:54.910006 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:54.910372 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:54.910757 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (916e2d16-6f68-4ffd-b4f6-b5d17c716574) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:54.911152 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.2, total_capacity_gb: 19.71,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:53.560267 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:54.912365 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:54.915215 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 364.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:54.918220 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:54.920296 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:54.922584 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:54.922584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:54.922584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:54.922584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.2, total_capacity_gb: 19.71,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:53.560267] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:54.922584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Weigher CapacityWeigher returned, weigher value is {max: 380.20000000000005, min: 380.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:54.923000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:54.923000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:54.923000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.2, total_capacity_gb: 19.71,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:54.915669 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:54.947910 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (beaf0ca6-e02c-4141-8dd0-14f1c5e61921) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:54.949893 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-824a9c71-4650-4330-81b3-fa32a79b457b tempest-VolumesListAdminTestJSON-1668309033 None] Flow 'volume_create_scheduler' (96fa9ee7-9de8-4d11-9bdb-cc18d77b93bd) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:55.915282 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1a07e12a-8306-4fcd-b7d6-7487feaab6ab None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:56.277187 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Flow 'volume_create_scheduler' (17c93827-d44c-417f-8514-218206d4ed46) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:56.279587 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6b195660-7ce4-4e60-a434-41b46e383066) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:56.281942 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6b195660-7ce4-4e60-a434-41b46e383066) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(399ca452-b668-4162-93bb-a9028ff1a5c9),volume_id=399ca452-b668-4162-93bb-a9028ff1a5c9,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:56.283845 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cf61d25b-38d8-4a18-b266-fc506381e00d) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:22:56.291089 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:22:56.306697 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 22, 55, 910606)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:22:56.307191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:22:56.307639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:56.308070 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (399ca452-b668-4162-93bb-a9028ff1a5c9) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:22:56.308539 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.71,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:55.910606 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:56.309679 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 344.0, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:22:56.311566 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:22:56.313730 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:22:56.316015 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:22:56.317824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:56.317824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:22:56.317824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.71,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:55.910606] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:22:56.317824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Weigher CapacityWeigher returned, weigher value is {max: 379.20000000000005, min: 379.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:22:56.318148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:22:56.318148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:22:56.318148 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.19, total_capacity_gb: 19.71,allocated_capacity_gb: 16, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 16,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:22:56.313070 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:22:56.370735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (cf61d25b-38d8-4a18-b266-fc506381e00d) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:22:56.372997 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-13a17724-e6c4-4cf6-b4f1-4d609a1224b8 tempest-VolumesListAdminTestJSON-1824872310 None] Flow 'volume_create_scheduler' (17c93827-d44c-417f-8514-218206d4ed46) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:22:57.995567 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6d261ca0-d841-4154-ac5f-36a9f9af2e83 tempest-VolumesListAdminTestJSON-1824872310 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 15, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:58.131502 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1d5b5162-5756-4f29-b5d9-4aed182dea88 tempest-VolumesListAdminTestJSON-1668309033 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:22:59.089813 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eae2624c-85bf-44d6-a53f-91c6875a91ea tempest-VolumesListAdminTestJSON-1668309033 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:00.153685 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7b47b717-e725-4912-9816-484842710bd6 tempest-VolumesListAdminTestJSON-1668309033 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:03.045811 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (28acb05a-72bb-434b-966b-4d562601ef22) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:03.048689 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cf562d89-d0b5-49dc-b3d6-1ff318d0d31b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:03.051097 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (cf562d89-d0b5-49dc-b3d6-1ff318d0d31b) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(c91a9a74-a7d0-4006-9f18-589c5f96cdf4),volume_id=c91a9a74-a7d0-4006-9f18-589c5f96cdf4,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:03.053017 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d6bcc6cd-8cec-4402-b837-2f5d70e191be) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:03.059869 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:03.073980 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 0, 149152)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:03.074412 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:03.074824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:03.075221 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (c91a9a74-a7d0-4006-9f18-589c5f96cdf4) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:03.075628 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:00.149152 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:03.077023 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:03.077422 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:03.077842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:00.149152] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:03.078269 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Weigher CapacityWeigher returned, weigher value is {max: 382.20000000000005, min: 382.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:03.078660 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:03.079135 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:03.079516 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.19, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:03.079164 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:03.105591 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (d6bcc6cd-8cec-4402-b837-2f5d70e191be) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:03.107708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-18863171-304a-4169-ba82-ed170f0beebb tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (28acb05a-72bb-434b-966b-4d562601ef22) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:17.959624 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-001c9e7c-ed13-4689-9d7f-78c15000dfbb tempest-VolumesBackupsTest-846679880 None] create_backup in rpcapi backup_id 3e7c431c-4c42-40e6-9c59-37dcad8a8e64 {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:23:28.058759 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-07cce73f-ed1c-4f67-9018-fa27728bcfc9 tempest-VolumesBackupsTest-846679880 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.18, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:28.490929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (72a49b59-3c66-46d3-b982-6d1e9de708f8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:28.493303 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (bbbe87ae-d0d2-47dd-938e-ec300c0977e1) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:28.495690 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (bbbe87ae-d0d2-47dd-938e-ec300c0977e1) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(8584d382-fe1d-46c6-b7b7-1345dee565bb),volume_id=8584d382-fe1d-46c6-b7b7-1345dee565bb,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:28.497561 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (28a84a5e-36d8-46cb-8506-b9242b4ee537) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:28.504178 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:28.517508 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.18, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 28, 54264)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:28.517929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:28.518293 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:28.518687 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (8584d382-fe1d-46c6-b7b7-1345dee565bb) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:28.519082 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.18, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:28.054264 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 383.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:28.520101 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:28.522866 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:28.525661 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 383.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:28.527614 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.18, total_capacity_gb: 19.71,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:28.054264] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:28.529708 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Weigher CapacityWeigher returned, weigher value is {max: 382.20000000000005, min: 382.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:28.530188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:28.530188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:28.530188 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.18, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:28.523048 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:28.548975 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (28a84a5e-36d8-46cb-8506-b9242b4ee537) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:28.551032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5dc38993-3d90-44db-ab42-4f6f1c7ca36e tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (72a49b59-3c66-46d3-b982-6d1e9de708f8) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:30.788150 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.backup.rpcapi [None req-635b8799-8414-41cf-a116-ec646e41d400 tempest-VolumesBackupsTest-846679880 None] create_backup in rpcapi backup_id 5d4e07e5-41a5-4ed5-b597-1a8352d12130 {{(pid=133283) create_backup /opt/stack/cinder/cinder/backup/rpcapi.py:58}} Dec 11 21:23:37.238800 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (c65458e6-7a4a-4445-a400-f1613d727b18) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:37.241270 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7ac5215e-b45b-4ff2-913f-8c78f130e72f) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:37.243824 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (7ac5215e-b45b-4ff2-913f-8c78f130e72f) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(99280ea9-5635-4ba2-a906-d30233819d72),volume_id=99280ea9-5635-4ba2-a906-d30233819d72,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:37.245726 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4324820f-c99e-43a7-bbff-5b26b822bbb5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:37.252690 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:37.266578 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.71, 'free_capacity_gb': 19.18, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 28, 54264)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:37.267000 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:37.267373 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:37.267735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (99280ea9-5635-4ba2-a906-d30233819d72) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:37.268129 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.18, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:28.523048 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:37.269169 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:37.271242 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 383.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:37.273066 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:37.275654 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.18, total_capacity_gb: 19.71,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:28.523048] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:37.277532 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Weigher CapacityWeigher returned, weigher value is {max: 381.20000000000005, min: 381.20000000000005} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:37.278080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:37.278080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:37.278080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.18, total_capacity_gb: 19.71,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:37.271539 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:37.298649 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (4324820f-c99e-43a7-bbff-5b26b822bbb5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:37.300911 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-4308be78-9a7b-4c96-922c-17529ceb905f tempest-VolumesBackupsTest-846679880 None] Flow 'volume_create_scheduler' (c65458e6-7a4a-4445-a400-f1613d727b18) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:46.047748 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c81603ce-6d70-4b78-b592-c8b6bfab18c7 tempest-VolumesBackupsTest-846679880 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.28, 'free_capacity_gb': 18.72, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:49.658688 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-de4f844b-ae14-4fda-a4fd-be64c1210a97 tempest-VolumesBackupsTest-846679880 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.25, 'free_capacity_gb': 18.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:52.045073 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Flow 'volume_create_scheduler' (ca0e5d7b-0ce7-419a-b013-ced5642e16b6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:52.049736 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c8b98cb2-49c7-4fd0-bea8-2012894f41dd) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:52.052472 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (c8b98cb2-49c7-4fd0-bea8-2012894f41dd) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(a6b986b9-a8a2-4f80-803b-73ba2db21b42),volume_id=a6b986b9-a8a2-4f80-803b-73ba2db21b42,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:52.055342 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (958b4dff-7cf0-42ef-a1f6-6a0be0854c99) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:52.066189 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:52.096580 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.25, 'free_capacity_gb': 18.27, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 49, 650877)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:52.097023 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:52.097389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:52.097745 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Checking if host ncdv-master-1@ceph#ceph can create a 2 GB volume (a6b986b9-a8a2-4f80-803b-73ba2db21b42) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:52.098139 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Checking provisioning for request of 2 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.25,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:49.650877 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:52.099211 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 363.6, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:52.101788 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:52.104170 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:52.107865 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:52.109848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:52.109848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:52.109848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.27, total_capacity_gb: 19.25,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:49.650877] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:52.109848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Weigher CapacityWeigher returned, weigher value is {max: 373.0, min: 373.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:52.109848 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:52.110261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:52.110261 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Consumed 2 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.27, total_capacity_gb: 19.25,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:52.102158 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:52.133322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (958b4dff-7cf0-42ef-a1f6-6a0be0854c99) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:52.135696 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-3769b9a4-8def-44ed-97fa-26e900612ff9 tempest-VolumesCloneNegativeTest-548181809 None] Flow 'volume_create_scheduler' (ca0e5d7b-0ce7-419a-b013-ced5642e16b6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:53.799956 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a0c1e64f-7ee1-44c0-b5cd-51a3ab242cb2 tempest-VolumesCloneNegativeTest-548181809 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.65, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:55.970258 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7d55c3ec-84fe-4531-adc2-710b7e932789 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.65, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:23:56.468123 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (4fd85672-2184-4cfb-922d-af283e261181) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:56.470515 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f7351b2e-8023-4489-bdfc-8b89babd9c64) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:56.472936 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (f7351b2e-8023-4489-bdfc-8b89babd9c64) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(5b5ab0b0-8c09-4846-8148-703817734fc8),volume_id=5b5ab0b0-8c09-4846-8148-703817734fc8,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:56.474877 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (53f7d158-2ee6-4982-9e07-1e877d6f3ab7) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:56.482338 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:56.498750 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.65, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 55, 965297)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:56.499172 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:56.499545 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:56.499917 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (5b5ab0b0-8c09-4846-8148-703817734fc8) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:56.500322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.11, total_capacity_gb: 19.65,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:55.965297 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:56.501729 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:56.502135 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:56.502533 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.11, total_capacity_gb: 19.65,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:55.965297] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:56.511451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Weigher CapacityWeigher returned, weigher value is {max: 381.0, min: 381.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:56.511451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:56.511451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:56.511451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.11, total_capacity_gb: 19.65,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:56.508656 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:56.537602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (53f7d158-2ee6-4982-9e07-1e877d6f3ab7) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:56.540134 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-889cb914-17f9-4e25-b41a-a76d991caa5b tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (4fd85672-2184-4cfb-922d-af283e261181) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:57.891142 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (e2079f0e-3cc8-42c9-94e8-0a4279d6b975) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:57.893568 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (be4726c8-8789-40e7-8075-babed5a767f4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:57.896021 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (be4726c8-8789-40e7-8075-babed5a767f4) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d848bd2d-5ba8-46ac-ae19-17ac452db5d6),volume_id=d848bd2d-5ba8-46ac-ae19-17ac452db5d6,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:57.898376 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b3c9a615-40c2-425b-b5d2-731d36819cd4) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:57.906579 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:57.927322 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.65, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 55, 965297)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:57.927735 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:57.928138 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:57.928531 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d848bd2d-5ba8-46ac-ae19-17ac452db5d6) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:57.929510 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.11, total_capacity_gb: 19.65,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:56.508656 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 365.4, 'size': 2, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:57.931096 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:57.933027 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:57.934671 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 382.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:57.936289 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:57.938089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:57.938089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:57.938089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.11, total_capacity_gb: 19.65,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:56.508656] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:57.938089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Weigher CapacityWeigher returned, weigher value is {max: 380.0, min: 380.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:57.938089 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:57.938429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:57.938429 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.11, total_capacity_gb: 19.65,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:57.933787 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:57.964373 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (b3c9a615-40c2-425b-b5d2-731d36819cd4) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:57.966513 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-9378cbfc-16df-46d5-84ff-4b9ea8a56d15 tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (e2079f0e-3cc8-42c9-94e8-0a4279d6b975) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:59.496852 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (2fb1de57-f31b-484e-9405-f174e0ba298e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:23:59.500614 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6d45f680-af71-45bb-a4ed-d780e9e7d010) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:59.509830 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (6d45f680-af71-45bb-a4ed-d780e9e7d010) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(fe54e817-43fa-4d2d-aee6-c3fc4b08e433),volume_id=fe54e817-43fa-4d2d-aee6-c3fc4b08e433,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:59.512476 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9eceb056-05a2-457d-9200-acc6b3c268d8) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:23:59.523841 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:23:59.538377 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.65, 'free_capacity_gb': 19.11, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 23, 55, 965297)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:23:59.539093 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:23:59.539624 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:59.540143 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (fe54e817-43fa-4d2d-aee6-c3fc4b08e433) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:23:59.540698 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.11, total_capacity_gb: 19.65,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:57.933787 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:59.541914 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:23:59.543776 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 362.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:23:59.545665 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:23:59.547577 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:23:59.549186 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:23:59.549186 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:23:59.551341 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:59.554339 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:23:59.554832 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.11, total_capacity_gb: 19.65,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:57.933787] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:23:59.555275 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Weigher CapacityWeigher returned, weigher value is {max: 379.0, min: 379.0} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:23:59.555639 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:23:59.556128 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:23:59.556518 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 16.11, total_capacity_gb: 19.65,allocated_capacity_gb: 15, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 15,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:23:59.556158 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:23:59.595005 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (9eceb056-05a2-457d-9200-acc6b3c268d8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:23:59.597520 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-288fe974-5dbc-4a68-8c96-285328dd35c6 tempest-VolumesListTestJSON-1032289019 None] Flow 'volume_create_scheduler' (2fb1de57-f31b-484e-9405-f174e0ba298e) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:24:04.148929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e8b47efd-d2ef-45cf-b594-81bd3c86c17f tempest-VolumesListTestJSON-1032289019 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.69, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 14, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:24:05.312073 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7c09f58e-8110-4f8d-ba90-566e4131b3bc tempest-VolumesListTestJSON-1032289019 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.69, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:24:06.383061 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f07c0abf-af3a-4e63-b8ab-5e8eb7900d58 tempest-VolumesListTestJSON-1032289019 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.69, 'free_capacity_gb': 19.16, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:24:45.853031 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c8a9f557-bc8e-4c2c-8894-89c2116be792 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:24:55.940759 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9d8b1715-c23d-44fd-a39b-6ef9c5ace2a8 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.68, 'free_capacity_gb': 19.13, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:25:36.724213 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.68, 'free_capacity_gb': 19.13, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 24, 55, 935520)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:25:36.724213 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:25:36.724862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:36.724862 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (None) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:25:36.725285 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.13, total_capacity_gb: 19.68,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:24:55.935520 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:25:36.726425 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:36.726810 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:36.727185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.13, total_capacity_gb: 19.68,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:24:55.935520] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:25:36.727617 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 381.6, min: 381.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:25:36.727969 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:25:36.728362 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-17566c50-0dfb-42cc-9deb-bd197e859d8c tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.13, total_capacity_gb: 19.68,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:25:36.728016 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:25:39.084701 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (1500bc90-b194-42ea-9766-3bfc6863bc06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:25:39.088065 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e42c1c5c-c72b-43e4-a797-8528384de75e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:25:39.090445 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (e42c1c5c-c72b-43e4-a797-8528384de75e) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend='ncdv-master-1@ceph#ceph',snapshot_id=d55a2bfd-57a0-4b9a-8f4f-7365cf7c4d7a,source_replicaid=,source_volid=None,volume=Volume(1252039d-ae8f-4317-9f1a-cad1b12ecd54),volume_id=1252039d-ae8f-4317-9f1a-cad1b12ecd54,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:25:39.092391 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fec3fe88-bc45-4da8-bd09-81e4bca2934b) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:25:39.099542 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:25:39.113791 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.68, 'free_capacity_gb': 19.13, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 24, 55, 935520)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:25:39.114185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:25:39.114583 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:39.114946 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (1252039d-ae8f-4317-9f1a-cad1b12ecd54) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:25:39.115362 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.13, total_capacity_gb: 19.68,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:25:36.728016 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:25:39.116372 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 342.2, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:25:39.118389 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 243, in create_snapshot Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: tgt_backend = self.driver.backend_passes_filters( Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 129, in backend_passes_filters Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:25:39.121282 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 382.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:25:39.123268 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.13, total_capacity_gb: 19.68,allocated_capacity_gb: 13, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 13,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:25:36.728016] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:25:39.125602 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Weigher CapacityWeigher returned, weigher value is {max: 380.6, min: 380.6} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:25:39.126124 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:25:39.126124 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:25:39.126124 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 17.13, total_capacity_gb: 19.68,allocated_capacity_gb: 14, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 14,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:25:39.118896 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:25:39.144540 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (fec3fe88-bc45-4da8-bd09-81e4bca2934b) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:25:39.146578 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-8c6d7228-6fba-4ca7-901b-237b5d2b08ec tempest-TestVolumeBootPattern-909971433 None] Flow 'volume_create_scheduler' (1500bc90-b194-42ea-9766-3bfc6863bc06) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:25:45.852622 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-45796240-c3f3-43e9-a7ef-af15b2af511e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:25:55.921178 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9590d9b8-6ebf-42df-a3af-52d0d3ccd338 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.68, 'free_capacity_gb': 19.13, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:26:45.856641 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-39cc2f13-ae94-4086-8a86-18c28340e166 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:26:55.921623 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ecb34f96-a59b-47b2-b663-b40bf12a42f0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.01, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 13, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:27:33.427564 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9e9e5271-b953-444e-b0d7-1b05ea7b502f tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.01, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:27:45.854386 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d231945-01a9-4191-82c8-7e64c6e62265 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:27:55.919191 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d9cdd116-77e5-46f3-b7bb-37ba4f8e9e70 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:28:10.801045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1eb56739-89cf-42d5-820f-5f03cfc21ffc tempest-TestVolumeBootPattern-909971433 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.09, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:28:45.858636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-eefbacf1-100a-4015-89d5-521390a20ac8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:28:55.939882 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3aa671c7-ac44-406a-868d-eae59aee99a5 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:29:45.863991 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4a8e20a5-0370-4504-8550-d30edbadb4a4 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:29:55.921507 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-79362e87-84d0-48be-ad8d-8dfe1849b10d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:30:45.860230 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2c8711fc-ebc9-4717-b50e-a071765bcde4 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:30:55.922022 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a78bd925-1ae3-41f6-979a-1eb332ead970 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:31:45.865405 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-07ddcfb1-f0f3-49d7-814b-b436e0ba31a2 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:32:45.869944 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-10fb86a1-7b7f-49d4-b0d8-a11d78c7b793 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:32:55.923407 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f04a3597-1851-4d96-895e-ad48739cc52b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:33:45.874668 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a2774a68-b6f4-479d-940c-9ae7aad5a37c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:33:55.940605 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ed91a226-f26f-46cc-8b42-1cdf02bef4f2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:34:45.879618 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-aa96b112-c761-401b-9408-1d5c5714ab09 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:34:55.919845 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cc2b8660-2205-4691-8965-2a31dc171875 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:35:45.884475 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9b115076-ad80-43f1-bce8-a0be39865a73 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:35:55.918542 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d5598391-fd39-49bc-9cad-d8ea2ae5abb0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:36:45.892050 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-a4a8382f-ad1f-4074-aa47-cf6d52032433 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:36:55.916990 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b09e1d91-bbf3-4dca-9911-5db9f8f23c8f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:37:45.889185 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-c21128c9-7340-482d-b503-66e7ea60c3f6 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:37:55.916245 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-26f355ce-7d55-4dcd-a7cc-2d01a82066cf None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:38:45.887389 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb7c64f5-b244-4d22-a7a0-a3cd3c587404 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:38:55.916454 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-77e60463-403c-403a-8279-064fc2fce5a7 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:39:45.890874 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-caa618cd-0d23-43ee-a89d-0097e81aa7e6 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:39:55.916458 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6a2351b1-03dc-4141-882d-8d225a99c5ed None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:40:36.035308 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Flow 'volume_create_scheduler' (5adc8591-0580-45c6-81c5-389369909dc6) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:40:36.037722 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8a73d998-3703-42a1-86fd-4c7a7bfe6d06) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:40:36.040137 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (8a73d998-3703-42a1-86fd-4c7a7bfe6d06) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(59dd43dd-716b-4188-98f7-66c9270749d7),volume_id=59dd43dd-716b-4188-98f7-66c9270749d7,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:40:36.042037 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (893a7fb2-14e4-4563-a28c-f9b38069e3fb) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:40:36.049950 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:40:36.063728 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 39, 55, 912380)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:40:36.064269 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:40:36.064641 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:40:36.065026 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (59dd43dd-716b-4188-98f7-66c9270749d7) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:40:36.065431 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:39:55.912380 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:40:36.066499 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:40:36.068475 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 362.59999999999997, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:40:36.070562 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:40:36.073162 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:40:36.075214 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:40:36.075214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:40:36.075214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:40:36.075214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:39:55.912380] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:40:36.075214 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Weigher CapacityWeigher returned, weigher value is {max: 381.8, min: 381.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:40:36.075669 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:40:36.075669 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:40:36.075669 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.19, total_capacity_gb: 19.64,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:40:36.068844 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:40:36.094414 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (893a7fb2-14e4-4563-a28c-f9b38069e3fb) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:40:36.096451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-5d33eac8-27b1-4654-bbc2-5a03d9146a47 tempest-ServerActionsTestJSON-1513721892 None] Flow 'volume_create_scheduler' (5adc8591-0580-45c6-81c5-389369909dc6) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:40:45.887058 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5117cae0-6720-4d26-b6ee-8e0345bb4f88 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:40:55.943768 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4d312da0-0286-42e8-b408-0a44d928b6e4 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:41:20.715259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3ef1fbaa-d433-448e-8e72-42b09c50cc55 tempest-ServerActionsTestJSON-1513721892 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:41:45.891690 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b2f96031-4de2-44f5-ae81-6608732ad6f8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:41:55.920054 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5db5fb31-0917-4a0a-92a0-5af87fa1a330 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:42:45.897402 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3ecf17a8-d2f4-4f2f-b765-732b5b97b03e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:42:54.439582 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Flow 'volume_create_scheduler' (b7a4f5d1-c404-491f-8eda-e005aaa5a186) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:42:54.442056 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (39315a9d-5ad7-499b-ac5f-662fe7ad9855) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:42:54.445115 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (39315a9d-5ad7-499b-ac5f-662fe7ad9855) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=None,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(4429c18d-e3f1-4a2a-87f2-aa5f120398c7),volume_id=4429c18d-e3f1-4a2a-87f2-aa5f120398c7,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:42:54.447116 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (81a4f1c5-d480-42ea-bf29-5674fec2caf5) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:42:54.454075 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:42:54.472996 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 41, 55, 915814)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:42:54.473413 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:42:54.473776 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:42:54.474132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (4429c18d-e3f1-4a2a-87f2-aa5f120398c7) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:42:54.474590 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:41:55.915814 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:42:54.475643 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: Message: 'Space information for volume creation on %(grouping)s %(grouping_name)s (requested / avail): %(requested)s/%(available)s' Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: Arguments: {'available': 383.8, 'size': 1, 'grouping': 'host', 'grouping_name': 'ncdv-master-1@ceph#ceph'} Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: --- Logging error --- Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: Traceback (most recent call last): Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: msg = self.format(record) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/handlers.py", line 156, in format Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: return logging.StreamHandler.format(self, record) + record.reset_color Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 929, in format Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: return fmt.format(record) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_log/formatters.py", line 500, in format Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: return logging.Formatter.format(self, record) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 668, in format Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: record.message = record.getMessage() Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: msg = msg % self.args Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: KeyError: 'requested' Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: Call stack: Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: func(*args, **kwargs) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_green.py", line 71, in __call__ Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: self.work.run() Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: res = self.dispatcher.dispatch(message) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: return self._do_dispatch(endpoint, method, ctxt, args) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: result = func(ctxt, **new_args) Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Dec 11 21:42:54.477938 ncdv-master-1 cinder-scheduler[133283]: return caller(func, *(extras + args), **kw) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/objects/cleanable.py", line 208, in wrapper Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: result = f(*args, **kwargs) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 87, in inject_operation_decorator Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: return schedule_function(*args, **kwargs) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/manager.py", line 229, in create_volume Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: flow_engine.run() Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: for _state in self.run_iter(timeout=timeout): Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 293, in run_iter Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: for transition in r.run_iter(builder.START): Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/automaton/runners.py", line 91, in run_iter Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: event = cb(old_state, new_state, event, *args, **kwargs) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 208, in schedule Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: not_done, failures = do_schedule(memory.next_up) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/builder.py", line 147, in do_schedule Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: return self._scheduler.schedule( Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 97, in schedule Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: futures.add(scheduler.schedule(atom)) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/scheduler.py", line 70, in schedule Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: return self._task_action.schedule_execution(task) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/actions/task.py", line 111, in schedule_execution Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: return self._task_executor.execute_task( Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 148, in execute_task Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: fut = self._executor.submit(_execute_task, Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 298, in submit Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: return self._gatherer.submit(fn, *args, **kwargs) Dec 11 21:42:54.480785 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 95, in submit Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: fut = self._submit_func(fn, *args, **kwargs) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 302, in _submit Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: self._run_work_func(_utils.WorkItem(fut, fn, args, kwargs)) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_futures.py", line 243, in Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: def __init__(self, green=False, run_work_func=lambda work: work.run()): Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/futurist/_utils.py", line 49, in run Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: result = self.fn(*self.args, **self.kwargs) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: result = task.execute(**arguments) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/flows/create_volume.py", line 147, in execute Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: self.driver_api.schedule_create_volume(context, request_spec, Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 99, in schedule_create_volume Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: backend = self._schedule(context, request_spec, filter_properties) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 555, in _schedule Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: weighed_backends = self._get_weighted_candidates(context, request_spec, Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filter_scheduler.py", line 365, in _get_weighted_candidates Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: backends = self.host_manager.get_filtered_backends(backends, Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/host_manager.py", line 530, in get_filtered_backends Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: return self.filter_handler.get_filtered_objects(filter_classes, Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 118, in get_filtered_objects Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: list_objs = list(objs) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/base_filter.py", line 45, in filter_all Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: if self._filter_one(obj, filter_properties): Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/__init__.py", line 29, in _filter_one Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: return passes_method(obj, filter_properties) Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: File "/opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py", line 169, in backend_passes Dec 11 21:42:54.483078 ncdv-master-1 cinder-scheduler[133283]: LOG.debug("Space information for volume creation " Dec 11 21:42:54.485731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:42:54.485731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:42:54.485731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:41:55.915814] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:42:54.485731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Weigher CapacityWeigher returned, weigher value is {max: 381.8, min: 381.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:42:54.485731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:42:54.486230 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:42:54.486230 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.19, total_capacity_gb: 19.64,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:42:54.477920 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:42:54.503870 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (81a4f1c5-d480-42ea-bf29-5674fec2caf5) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:42:54.506470 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-248d24bb-74a5-4abb-8dc8-888d924dde81 tempest-TestMinimumBasicScenario-1618858629 None] Flow 'volume_create_scheduler' (b7a4f5d1-c404-491f-8eda-e005aaa5a186) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:42:55.930866 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1511b04c-b7f8-4c07-be95-d21f3b5721ce None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:43:45.903334 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-818ebd1d-e9b9-4a61-8023-987beb592848 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:43:55.914636 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-be4016ee-0f73-453a-b573-7e3e89ecf954 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:44:15.540090 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3997934a-8cf5-48a2-8ac5-9c66f9ca3465 tempest-TestMinimumBasicScenario-1618858629 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:44:45.901080 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e551b614-67f5-41be-af5a-5b36a5de013c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:44:55.913813 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7777de89-2d07-4a77-90d3-b3d56534c377 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:45:45.904692 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e89a8940-ecf7-426d-a199-e6325b6883dc None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:45:55.918575 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dfdf2e4c-9232-4351-ada9-33618ed3f65b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:46:45.909762 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-36060de5-7e8e-4570-95b0-5971871f8af9 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:46:55.918239 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ee5400c7-bb52-4f73-85ce-5ea94132631d None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:47:45.914584 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b0b3c638-14fb-42c3-9325-08b8fa3514e9 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:47:55.929675 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2fb1d15b-0532-49eb-b546-91a37cfcb03b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:48:45.919305 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-ff672f18-a9b2-4e1d-bad3-fd511081275e None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:48:55.920521 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-dab37305-4754-4fcb-885f-c8d0f082509f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:49:45.923929 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-64b73b19-20a5-4d2d-b53b-55909a7b9ef5 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:49:55.920130 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5b32a07e-491c-4f09-90e1-6ef044b774b2 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:50:45.928588 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7e208653-3729-48a2-ae87-dd18dd4c1804 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:50:55.919875 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-07f77ff5-6c08-46ab-a02e-cc54b56f5687 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:51:45.925187 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb2dd8d0-7a9f-4823-a557-a3e5cec24e79 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:51:55.918260 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e703776b-a491-4931-9709-5a9bf9d926da None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:52:45.929868 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-205b48fb-7f0b-42f9-9b05-3d43123c4766 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:52:55.917512 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-f3284b0f-1b99-4b8b-b047-50967e7c4554 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:53:45.936390 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-333878db-aaed-49e3-901d-a0c9ab43bd18 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:53:55.915382 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e5024bb2-1e51-48b7-ac66-ad1b1cb62306 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:54:45.933575 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-042529f7-5478-4876-9c00-7aa11ce0242c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:54:55.940542 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3e9e0bb6-889f-4256-acdf-a0aa3718b63b None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:55:45.929890 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-279c9215-5569-4c5b-a71d-38e8b75ec38c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:55:55.926538 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0b5a090b-4713-4b72-a40c-05c906b773bb None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:56:31.290720 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Flow 'volume_create_scheduler' (29a779d3-6127-48c0-ba61-2f32816b8211) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:56:31.293302 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (79b2d1b9-9164-44d1-bba5-c1be1789ce65) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:56:31.295743 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Task 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create' (79b2d1b9-9164-44d1-bba5-c1be1789ce65) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'request_spec': RequestSpec(CG_backend=,availability_zones=['nova'],backup_id=None,cgsnapshot_id=None,consistencygroup_id=None,group_backend=,group_id=None,image_id=659d8dfa-8ba1-483c-93c0-647553900999,operation=,resource_backend=,snapshot_id=None,source_replicaid=,source_volid=None,volume=Volume(d09a9fe3-9a30-44e0-9d8f-f30bf776750a),volume_id=d09a9fe3-9a30-44e0-9d8f-f30bf776750a,volume_properties=VolumeProperties,volume_type=VolumeType(5aaae771-290d-40aa-8e6c-3fda92dceecd))}' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:56:31.297659 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bfafc9c3-2a53-450e-bf1f-dab0bf3822ae) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:190}} Dec 11 21:56:31.305262 ncdv-master-1 cinder-scheduler[133283]: DEBUG oslo_db.sqlalchemy.engines [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=133283) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 11 21:56:31.321032 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Updating capabilities for ncdv-master-1@ceph#ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None, 'timestamp': datetime.datetime(2021, 12, 11, 21, 55, 55, 920856)} {{(pid=133283) update_from_volume_capability /opt/stack/cinder/cinder/scheduler/host_manager.py:392}} Dec 11 21:56:31.321444 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Starting with 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:97}} Dec 11 21:56:31.321807 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:56:31.322162 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Checking if host ncdv-master-1@ceph#ceph can create a 1 GB volume (d09a9fe3-9a30-44e0-9d8f-f30bf776750a) {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:58}} Dec 11 21:56:31.322582 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filters.capacity_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Checking provisioning for request of 1 GB. Backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:55:55.920856 {{(pid=133283) backend_passes /opt/stack/cinder/cinder/scheduler/filters/capacity_filter.py:133}} Dec 11 21:56:31.324045 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Filter CapacityFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:56:31.324446 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_filter [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=133283) get_filtered_objects /opt/stack/cinder/cinder/scheduler/base_filter.py:125}} Dec 11 21:56:31.324806 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Filtered [host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 19.19, total_capacity_gb: 19.64,allocated_capacity_gb: 11, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 11,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:55:55.920856] {{(pid=133283) _get_weighted_candidates /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:370}} Dec 11 21:56:31.325218 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.base_weight [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Weigher CapacityWeigher returned, weigher value is {max: 381.8, min: 381.8} {{(pid=133283) get_weighed_objects /opt/stack/cinder/cinder/scheduler/base_weight.py:154}} Dec 11 21:56:31.325561 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Weighed [WeighedHost [host: ncdv-master-1@ceph#ceph, weight: 0.0]] {{(pid=133283) get_weighed_backends /opt/stack/cinder/cinder/scheduler/host_manager.py:542}} Dec 11 21:56:31.326025 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.filter_scheduler [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Choosing ncdv-master-1@ceph#ceph {{(pid=133283) _choose_top_backend /opt/stack/cinder/cinder/scheduler/filter_scheduler.py:605}} Dec 11 21:56:31.326397 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Consumed 1 GB from backend: host 'ncdv-master-1@ceph#ceph':free_capacity_gb: 18.19, total_capacity_gb: 19.64,allocated_capacity_gb: 12, max_over_subscription_ratio: 20.0,reserved_percentage: 0, provisioned_capacity_gb: 12,thin_provisioning_support: True, thick_provisioning_support: False,pools: {},updated at: 2021-12-11 21:56:31.326053 {{(pid=133283) consume_from_volume /opt/stack/cinder/cinder/scheduler/host_manager.py:343}} Dec 11 21:56:31.353557 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Task 'cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create' (bfafc9c3-2a53-450e-bf1f-dab0bf3822ae) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=133283) _task_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:178}} Dec 11 21:56:31.355565 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.manager [None req-77f8abb7-10df-4e32-8259-415dee130b60 tempest-TestShelveInstance-505849774 None] Flow 'volume_create_scheduler' (29a779d3-6127-48c0-ba61-2f32816b8211) transitioned into state 'SUCCESS' from state 'RUNNING' {{(pid=133283) _flow_receiver /usr/local/lib/python3.8/dist-packages/taskflow/listeners/logging.py:141}} Dec 11 21:56:45.935998 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-95a3954a-4ed5-4379-829d-d2de664826f8 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:56:55.930418 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bc83bd48-d35e-4522-b0db-bee29f231588 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.64, 'free_capacity_gb': 19.19, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:57:45.939241 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1261fc83-4632-4723-889c-8f21670dfa4d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:57:55.929997 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-186c5452-a116-4520-9ebc-ca8f9317f164 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.62, 'free_capacity_gb': 19.1, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:58:45.944392 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4907f578-b587-4698-ad26-7e3969c0fa3c None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:58:55.923996 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7a6a0c5d-d652-4af8-a288-bea2361d5c8f None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.61, 'free_capacity_gb': 19.07, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 21:59:45.951731 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-155731b2-2d64-45f7-bbbd-010ecb0463c7 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 21:59:55.930510 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-92676cfe-befd-430a-979e-bb1d8ae64b25 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.05, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 12, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:00:45.949789 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-6f3f55a3-e576-43bd-a4f7-77ab760e281d None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:00:46.547817 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [req-5a464727-3575-4b11-856a-1a1cc68b62b5 req-52d36e29-2d9a-4082-96d3-31ca54deefdd tempest-TestShelveInstance-505849774 None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.05, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:00:55.965398 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-fb908a61-6677-40ce-9b81-562d4c17d5a0 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:01:45.952182 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0e5a5876-b2ba-4715-8b24-58d6eb875f47 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:01:55.923722 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-62d5444d-9a3d-448b-94c1-f56eb98beaf9 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:02:45.957524 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-17c31a3f-95fb-42bd-9884-2475cd8ba7ec None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:02:55.923603 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-9f883df8-becb-4fbc-a08c-59b85d76bdf6 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:03:45.961970 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-325a167e-1332-4fd8-a15d-e264420641c6 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:03:55.933465 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-53c5512d-4f3c-4fab-9077-fe2bfffccc09 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:04:45.967259 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2078d5fe-eb52-4dbc-9db5-9d29d8a5e5e2 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:04:55.928491 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7cae0e59-73d8-4db3-949c-dfa9708d32ab None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:05:45.972195 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-e9679bc1-69c9-43a0-8e30-16b88a8aae64 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:05:55.917680 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-3d465c86-baf6-4c2e-bebe-264f308e4949 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:06:45.977110 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-800a51b7-a166-4c67-95a7-afe9c694a49b None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:06:55.916274 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5f0c2140-8734-4a11-be59-ee7af2b39ec1 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:07:45.982323 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-cb5e8b1a-5f82-4cbd-aa48-47d81ff250d5 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:07:55.916132 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-beba1715-c3e4-4f9e-894f-118f79bd27fa None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:08:45.987842 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b6fcc928-4a32-4144-8cae-e12c05cc8e59 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:08:55.916197 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-1958f056-e8bd-49d6-bf5c-c0ef736b93fc None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:09:45.993021 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-696a3583-5037-4983-a964-6ce0c3f7c1b6 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:09:55.915394 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-bab48205-13c3-4ee1-9bc7-9afefc8f1b1c None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:10:45.998451 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-4fd30e92-74ec-4d38-a8c9-ddc83f703311 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:10:55.915808 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-689782d0-f39c-46de-afb0-17764593f922 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:11:46.003314 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-d80f8362-744b-4244-abfc-71f9dd0714fb None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:11:55.914765 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-5223759e-a0ef-4f76-a950-18653945c197 None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:12:45.999258 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-0b962e4e-91d2-4321-b199-24af3a71c5e4 None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:12:55.923189 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-7339527a-3639-4d68-8f38-70b40470104e None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:13:46.007055 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-b33b281d-35dd-41b0-9518-7fa76f05911a None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}} Dec 11 22:13:55.915729 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-2e6fc721-e48e-4081-90fa-bc7266b1eebf None None] Received volume service update from ncdv-master-1@ceph: {'vendor_name': 'Open Source', 'driver_version': '1.2.0', 'storage_protocol': 'ceph', 'total_capacity_gb': 19.6, 'free_capacity_gb': 19.15, 'reserved_percentage': 0, 'multiattach': True, 'thin_provisioning_support': True, 'max_over_subscription_ratio': '20.0', 'location_info': 'ceph:/etc/ceph/ceph.conf:6e8f856c-f0dd-4f2b-b584-866af3a1c6ae:cinder:volumes', 'backend_state': 'up', 'volume_backend_name': 'ceph', 'replication_enabled': False, 'allocated_capacity_gb': 11, 'filter_function': None, 'goodness_function': None} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:605}} Dec 11 22:14:46.009986 ncdv-master-1 cinder-scheduler[133283]: DEBUG cinder.scheduler.host_manager [None req-00d2d631-fecb-4d40-b833-c8198565c2ca None None] Received backup service update from ncdv-master-1: {'backend_state': True, 'driver_name': 'cinder.backup.drivers.ceph.CephBackupDriver', 'availability_zone': 'nova'} {{(pid=133283) update_service_capabilities /opt/stack/cinder/cinder/scheduler/host_manager.py:574}}